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FOREWORD 


This  report  presents  the  results  of  research  conducted  to  test  the  effec¬ 
tiveness  of  a  preliminary  design  for  a  laboratory  to  be  used  for  command  and 
control  (C^)  behavioral  research.  This  laboratory  facility  is  being  developed 
at  ART'S  Fort  Leavenworth,  Kansas,  Field  Unit.  The  facility,  known  as  the 
Experimental  Development,  Demonstration,  and  Integration  Center  (EDDIC) ,  will 
be  used  to  evaluate  the  human  aspects  of  concepts  and  designs  for  future  Army 
systems.  The  experimental  design  used  in  this  research  is  being  modified 
and  upgraded  in  the  F.DDIC  to  provide  state-of-the-art  computer  hardware  and 
software,  and  to  facilitate  multi -subject,  multi- function  experimentation  and 
the  integration  and  evaluation  of  the  latest  technology  in  decision  support 
systems.  The  experimental  results  reported  here  also  provide  insight  into  the 
use  of  information. in  course  of  action  development  by  tactical  decision  mak¬ 
ers.  This  understanding  is  necessary  for  the  development  of  acceptable  and 
effective  decision  support  systems. 


EDGAR  M.  JOHNSON 
Technical  Director 
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EFFECTS  OF  EXPERTISE  AND  COGNITIVE  STYLE  ON  INFORMATION  USE 
IN  TACTICAL  DECISION  MAKING 


EXECUTIVE  SUMMARY 


Requirement; 

To  investigate  the  effects  of  expertise,  cognitive  style,  and  mission  on 
what  information  military  officers  use  in  tactical  decision  making  problems 
and  how  it  contributes  to  decision  making  processes.  Also,  to  evaluate  a 
method  of  decision  making  research  involving  automated  presentation  and  re¬ 
sponse  recording. 


Procedure : 

Sixteen  military  officers  were  each  given  two  tactical  problems  to  solve 
involving  the  development  of  a  concept  of  operation  for  an  offensive  and  a  de¬ 
fensive  mission.  The  information  available  to  them  was  typical  of  that  avail¬ 
able  at  a  division  command  post  except  that  it  was  presented  by  a  computer 
system  in  both  graphic  and  textual  form.  The  system  was  designed  to  require 
the  subject  to  take  notes  from  the  basic  data  to  develop  the  required  decision 
products.  Participants  were  allowed  to  select  any  information  to  view. 

Eight  of  the  subjects  were  lieutenant  colonel  instructors  and  eight  were 
majors  who  were  students  at  the  Command  and  General  Staff  College,  Fort 
Leavenworth.  Each  subject  was  given  the  Embedded  Figures  Test  to  measure 
cognitive  style.  A  post-experiment  questionnaire  was  used  to  determine  the 
subjects'  opinions  on  various  aspects  of  the  experimental  design. 


Findings : 

a.  The  instructors  used  less  information  than  did  the  students  and  the 
information  used  by  instructors  consisted  of  more  summary  information  and  less 
detailed  information  than  that  used  by  the  students. 

b.  There  was  no  relationship  between  the  Embedded  Figures  Test  scores 
and  the  measures  of  information  use . 

c.  There  was  no  significant  effect  of  mission  type  on  information  use. 

d.  There  was  only  a  very  general  sequential  viewing  pattern  of  infor¬ 
mation  use  across  subjects;  individual  differences  were  prevalent. 

e.  The  subjects  generally  felt  that  this  was  a  worthwhile  means  of  tac¬ 
tical  decision  making  research  but  they  had  many  suggestions  for  improvements. 
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Utilization  of  Findings: 


The  findings  are  useful  in  the  design  of  automated  command  and  control 
systems  because  they  suggest  that  such  systems  should  be  adaptable  to  indi¬ 
vidual  differences  and  that  the  design  of  such  systems  should  be  based  upon 
inputs  from  multiple  users.  The  effects  of  expertise  suggest  that  only  the 
actual  user  population  should  establish  requirements  for  tactical  decision 
aids.  The  findings  concerning  the  research  design  are  being  used  in  designing 


ARI's  Experimental  Design 
Leavenworth  for  work  on  C 


Demonstration,  and  Integration  Center  at  Fort 
automation  issues  such  as  distributed  decision 


making,  the  development  of  planning  aids,  the  effects  of  automation  on  or¬ 


ganizational  structure,  and  training  for  automation. 
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EFFECTS  OF  EXPERTISE  AND  COGNITIVE  STYLE 
ON  INFORMATION  USE  IN  TACTICAL  DECISION  MAKING 


INTRODUCTION 

It  is  becoming  increasingly  clear  that  automated  decision  aids  and  informa¬ 
tion  systems  for  Army  command  and  control  (C^)  cannot  be  implemented  and  used 
in  isolation  (Noah  &  Halpin,  1986;  Riedel,  1987).  They  must  be  part  of  a  broad 
integrated  system  with  a  data  base  common  to  a  number  of  functional  areas  and 
with  a  group  of  automated  aids  that  can  support  a  variety  of  complex  tasks 
within  those  areas.  The  use  of  single  isolated  task  specific  aids  would  impose 
too  great  a  burden  on  the  users.  The  user  would  be  required  to  stop  at  the 
appropriate  step  in  a  larger  task,  remember  the  aid,  remember  how  to  access  and 
use  the  aid,  and  transfer  the  results  to  Che  next  step  in  the  problem.  A  broad 
integrated  system  should  be  capable  of  aiding  multiple  users  in  multiple  tasks 
in  multiple  environments.  This  means  that  such  a  system  must  accommodate  to  a 
wide  variety  of  user  needs,  skill  and  ability  levels,  expertise  and  prefer¬ 
ences.  A  critical  part  of  building  such  an  adaptive  system  is  determining  what 
user  characteristics  the  system  should  accommodate  and  how  it  should  do  this. 

In  order  to  guide  the  design  of  adaptive  decision  support  systems,  a  program  of 
research  is  needed  that  investigates  the  effects  of  individual  differences  on 
performance  using  military  decision  support  systems.  One  objective  of  the 
research  reported  here  is  to  provide  some  preliminary  insights  into  the  degree 
and  patterns  of  individual  differences  in  information  requirements  in  represen¬ 
tative  tactical  situations.  The  individual  differences  of  expertise  and  cog¬ 
nitive  style  were  chosen  for  investigation  because  non-military  research 
literature  shows  these  variables  affect  decision  making  procedures. 

A  second  objective  of  this  study  was  to  design,  implement,  and  test  a 
method  of  determining  information  use  via  a  computerized  system  of  information 
presentation  and  recording.  This  objective  stems  from  problems  in  determining 
information  requirements  and  measuring  information  usage.  In  order  to  study 
individual  differences  in  information  requirements,  a  reliable  and  valid  method 
of  collecting  data  on  information  requirements  is  needed. 

The  approach  to  studying  information  requirements  that  is  implemented  in 
this  experiment  uses  a  laboratory  setting  to  examine  actual  information  usage. 
Past  attempts  to  define  the  information  requirements  of  commanders  and  staffs 
have  most  often  simply  asked  decision  makers  what  information  that  they  needed 
or  used  in  their  decision  making.  This  method  often  produces  lists  of  desired 
information  too  large  to  be  accommodated  by  a  battlefield  support  system  or  the 
resulting  requirements  are  ill-defined  and  not  related  to  situational  varia¬ 
bles.  Another  problem  is  that  decision  makers  often  do  not  know  what  informa¬ 
tion  they  actually  need,  but  infer  this  from  the  information  that  standard 
military  procedures  mandate  they  use.  Defining  information  requirements  by  ac¬ 
tual  usage  circumvents  both  of  these  problems. 
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One  advantage  of  a  laboratory  approach  is  that  it  allows  for  control  of 
extraneous  variables  such  that  the  relationships  between  the  variables  of  in¬ 
terest  can  be  measured.  Thus  we  can  use  the  laboratory  to  present  realistic 
tactical  problems  to  Army  officers,  each  officer  receiving  exactly  the  same 
problem  under  the  same  conditions.  We  can  present  available  battlefield  infor¬ 
mation  through  a  computer  system  and  require  that  the  officer  respond  via  the 
computer,  thus  allowing  us  to  record  all  that  he  views  and  does.  We  can  meas¬ 
ure  how  the  officers  differ  in  what  information  they  use  and  how  they  use  it 
and  see  if  this  relates  in  any  way  with  differences  in  their  experience,  back¬ 
ground,  and  personal  style  of  decision  making.  We  can  also  present  each  officer 
with  a  variety  of  tactical  situational  variables.  A  laboratory  approach  per¬ 
mits  a  means  of  determining  what  tactical  information  is  used,  how  much  it  is 
used,  and  when  it  is  used.  Also  it  allows  the  determination  of  the  relation¬ 
ships  among  information  items.  This  is  of  great  value  in  designing  the  data 
combinations  and  display  formats  within  a  command  and  control  (C^)  support 
system.  The  intent  of  such  a  research  approach  is  not  to  replace  other  system¬ 
atic  means  of  investigating  information  requirements,  but  rather  to  supple¬ 
ment  them  with  an  approach  that  provides  unique  insights  and  helps  validate  the 
findings  of  other  methods. 


The  results  of  the  experiment  reported  here  will  be  used  to  help  guide  the 
design  of  a  permanent  laboratory  system.  This  laboratory  is  intended  for  re¬ 
search  into  Army  processes  and  in  the  development,  demonstration  and  inte¬ 
gration  of  computer  aiding  into  future  Army  systems. 

2 

The  results  are  also  valuable,  to  C  system  designers,  especially  those 
concerned  with  operations  (G3)  subsystem  design  at  division  and  corps  level. 

For  instance,  findings  concerning  the  general  sequential  pattern  of  information 
use  in  operational  planning  and  the  problems  encountered  with  the  use  of  com¬ 
puter  graphics  should  be  considered  in  effective  system  design.  Also,  the 
discovery  of  significant  group  differences  in  information  use  is  a  precaution¬ 
ary  note  to  system  designers  in  how  they  go  about  establishing  system  require¬ 
ments.  Many  other  insights  might  be  drawn  from  the  reported  findings  related 
to  specific  categories  of  information  and  their  value  to  the  operational  plan¬ 
ning  process. 

To  summarize,  Che  general  objective  for  the  exploratory  investigation  re¬ 
ported  here  was  to  provide  some  preliminary  insights  into  the  degree  and  pat¬ 
terns  of  individual  differences  in  decision  making  in  tactical  situations  that 
are  representative  of  those  in  a  combat  environment.  Also  of  interest  was  the 
suitability  of  an  approach  to  determine  information  use  via  a  computerized 
system  of  information  presentation  and  recording. 


2 


BACKGROUND 


Information  Requirements  and  Usage 

One  problem  that  affects  the  design  of  information  and  decision  support 
systems  is  that  information  requirements  are  not  invariant  over  the  users.  We 
need  to  know  to  what  extent  an  optimal  information  system  should  accommodate 
individual  preferences  for  different  types,  formats,  and  levels  of  aggregation 
of  information.  However,  to  study  individual  differences  in  information  usage, 
we  need  a  valid  methodology  for  determining  information  requirements.  In  the 
present  study  we  designed,  implemented,  and  tested  a  methodology  to  determine 
information  requirements. 

There  are  several  approaches  that  can  be  used  to  define  the  information 
requirements  of  commanders  and  staffs.  An  analysis  of  the  decision  task 
could  yield  a  list  of  information  requirements  that  would  logically  and  doc- 
trinally  be  needed.  A  second  approach  is  to  ask  expert  tactical  decision  mak¬ 
ers  what  information  is  important  to  their  decisions.  One  problem  with  this 
approach  is  that  expert  decision  makers  sometimes  cannot  accurately  specify  the 
basis  on  which  they  make  their  decisions  (Ericsson  &  Simon,  1980;  &  Nisbett  & 
Wilson,  1977).  In  addition,  subjective  opinions  are  influenced  by  how  the 
questions  are  asked.  A  third  methodology,  and  the  one  used  in  this  study,  is 
to  examine  the  information  decision  makers  actually  use  in  a  decision  making 
task.  This  methodology  uses  computer  monitored  data  collection  of  information 
usage  in  a  laboratory  setting.  This  technique  has  a  number  of  advantages. 

Data  collection  is  unobtrusive,  thereby  reducing  threats  to  validity  like  de¬ 
mand  characteristics  and  interviewer  bias.  The  data  are  not  based  on  memory  or 
reports  of  behavior  but  on  actual  behavior.  It  is  as  easy  to  capture  the  en¬ 
tire  set  of  behaviors  as  it  is  to  sample.  Thus  one  avoids  measurement  and 
reliability  problems  created  by  questionnaire  wording,  sampling  error,  and 
response  inaccuracy. 

There  are  several  disadvantages  to  defining  information  requirements  by 
information  usage.  If  the  type  of  behavior  being  measured  is  fairly  objective, 
summarization  and  analysis  of  the  data  can  also  be  done  by  the  computer.  How¬ 
ever,  if  subjective  techniques  such  as  protocol  analyses  (Ericsson  &  Simon, 
1984)  are  used,  the  analysis  may  be  very  time  consuming  and  expensive.  Another 
disadvantage  with  determining  what  information  is  actually  being  used  for  deci¬ 
sion  making  and  then  building  this  information  into  the  decision  support  system 
is  that  this  approach  will  not  necessarily  lead  to  improved  performance.  Aids 
may  be  good  at  eliminating  inconsistencies  in  users'  application  of  their  in¬ 
formation  selection  strategies,  but  the  aid  cannot  correct  systematic  biases. 
That  is,  the  current  pattern  of  usage  may  be  suboptimal  and  automation  alone 
will  not  overcome  the  problem.  A  profitable  strategy  to  determine  information 
needs  would  be  based  on  a  combination  of  doctrine,  task  analysis,  past  behav¬ 
ior,  current  usage,  and  prescriptive  decision  making  behaviors. 


3 


Individual  Differences 


Another  objective  of  this  stuay  was  to  examine  the  effects  of  two  individ¬ 
ual  difference  variables  on  decision  making  performance.  Research  shows  that  a 
variety  of  individual  difference  factors  can  greatly  influence  how  individuals 
use  and  process  information  (Robertson  &  Meshkati,  1985).  If  the  decision 
support  system  is  to  be  at  least  as  effective  as  unaided  decision  making  it 
should  accommodate  individual  preferences  in  information  usage  and  aggrefcation. 
Reviewing  the  literature  on  individual  differences,  Sage  (1981)  concluded  that 
all  available  evidence  suggests  that  it  is  necessary  to  incorporate  the  deci¬ 
sion  makers’  characteristics  into  Che  design  of  Che  decision  support  system. 

As  studies  of  the  design  of  automated  decision  aids  and  information 
systems  mature,  further  advances  may  depend  on  developments  in  Che  area  of 
individual  differences  in  decision  making.  There  are  two  related  reasons  for 
this.  First,  virtually  every  paper  on  designing  aids  and  information  system 
admonishes,  "involve  the  user  in  aid  design."  The  problem  facing  designers  is 
which  user  to  involve.  When  aids  were  designed  only  for  specific  tasks  and  had 
a  limited  range  of  users,  Che  question  of  who  the  intended  user  was  had  an  easy 
answer.  However,  it  is  probable  Chat  the  aids  being  developed  today  will  be 
part  of  a  system  serving  multiple  users.  Further,  the  use  of  the  system  may 
be  extended  in  the  future  to  now  unknown  classes  of  users.  It  will  be  impossi¬ 
ble  to  obtain  a  representative  from  all  classes  of  possible  users.  On  the 
other  hand,  the  aid  design  should  be  appropriate  for  all  individual  users. 
Designers  need  to  know  which  user  characteristics  can  be  ignored  in  selecting  a 
user  and  which  ones  should  impact  the  design  of  the  aid.  Does  it  natter 
whether  the  user  is  a  captain  or  a  major?  Does  it  matter  whether  he  or  she 
belongs  to  one  tactical  unit  or  another?  Does  it  matter  whether  the  user  is  25 
or  45  years  old?  The  aid  designers  must  be  able  to  specify  to  which  user  popu¬ 
lation  the  results  of  their  studies  with  the  user  can  be  generalized.  The 
study  of  individual  differences  in  information  processing  and  decision  making 
can  help  define  which  user  characteristics  should  be  considered  in  choosing  a 
user  for  design  studies. 

A  second  reason  why  the  study  of  individual  differences  is  important  to  the 
design  of  decision  aids  is  related  to  the  integrated  nature  of  future  aids.  As 
discussed  previously,  it  is  likely  that  the  decision  aids  and  information  sys¬ 
tems  of  the  future  will  be  part  of  a  large  scale  decision  support  system  which 
will  satisfy  multiple  functions  for  a  variety  of  users  with  a  variety  of  per¬ 
sonal  characteristics.  As  a  means  of  accommodating  many  tasks,  environments, 
and  users  within  the  same  decision  support  system,  a  number  of  writers  (e.g., 
Lehner,  Cohen,  Mullin,  Thompson,  &  Laskey,  1987;  Noah  &  Halpin,  1986;  Saga, 
1985),  have  suggested  the  concept  of  the  adaptive  aid  or  adaptive  user  inter¬ 
face.  The  user/system  interface  of  these  proposed  systems  generally  include  a 
number  of  modules: 

1.  User's  model  of  the  task. 

2.  System's  model  of  the  task. 

3.  User's  raodel  of  the  user. 
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4.  System's  model  of  the  user. 

5.  User's  model  of  the  system. 

6.  System's  model  of  the  system. 

Both  the  user's  and  system's  models  of  the  user  require  that  the  user 
characteristics  incorporated  into  the  model  be  specified.  At  this  time  the 
literature  on  user  characteristics  does  not  provide  a  good  basis  for  specifying 
what  user  variables  should  be  included.  Research  on  the  effects  of  individual 
differences  on  decision  quality  can  help  determine  which  user  variables  are 
important  and  should  be  included  in  these  models. 

Another  interface  module,  not  included  in  the  list  above  but  which  is 
implied  in  descriptions  of  how  the  interface  functions,  is  a  catalog  which 
matches  possible  user  characteristics,  or  patterns  of  characteristics,  with 
optimal  aid  characteristics.  The  aid's  model  of  the  user  activates  certain 
characteristic  values  within  the  catalog  which  in  turn  activate  the  appropriate 
set  of  aid  characteristics.  Construction  of  this  catalog  again  requires  a 
knowledge  of  which  characteristics  have  a  significant  impact  on  decision  qual¬ 
ity.  In  addition,  information  about  which  aid  characteristics  are  appropriate 
for  which  user  characteristic  is  needed.  At  the  present,  we  have  little  knowl¬ 
edge  about  such  remedial  design  characteristics.  Extensive  research  on  the 
effects  of  individual  differences  on  information  processing  and  decision  making 
is  needed  before  such  a  catalog  is  feasible. 

Individual  Differences  Research 

This  section  briefly  outlines  an  approach  to  research  the  effects  of  indi¬ 
vidual  differences  on  decision  making.  Such  an  approach  could  provide  useful 
input  to  the  design  of  an  adaptive  aid  or  interface. 

Any  characteristics,  on  which  potential  users  vary  and  which  might  affect 
information  processing  and/or  decision  making  performance,  are  candidates  for 
investigation.  The  following  characteristics  have  been  suggested  in  the  litera¬ 
ture.  Each,  except  gender,  could  be  a  between  user  variable  or  within  user 
variable.  The  latter  could  vary  across  time,  task,  or  environmental  situation. 

Demographic  Characteristics 

Age 

Gender 

Rank/Conmand  Level 
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Personality  Characteristics 
Decision  making  style 
Cognitive  style 
Learning  style 
Risk  taking  propensity 
Motivation 
Locus  of  control 
Ski 11s /Abilities 
Task  expertise 

Knowledge  of  task  requirements 
Skill/experience  with  the  system 
Training 
Spatial  ability 
Intelligence 
Preferences 


Goals 

Preferences  for  display  format 
Preferred  sensory  modality 
Preferred  communication  mode 

Many  of  those  characteristics  probably  interact  with  the  task  and  environ¬ 
ment  to  determine  information  processing  and  decision  making  performance,  and 
the  characteristics  should  not  be  studied  without  considering  task  and  environ¬ 
ment.  For  example,  the  novice  may  be  able  to  perform  as  well  as  the  expert 
given  a  low  stress  situation,  but  not  in  a  high  stress  situation. 

General  performance  variables  to  consider  in  studying  individual  differ¬ 
ences  might  include: 

Ease  of  use  of  the  aid 

Ease  of  learning  of  the  aid 
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User  acceptance 
Quality  of  the  decision 
Speed  of  decision  caking 
Patterns  of  information  usage 

Aid/system  characteristics  which  could  be  adapted  to  acconnodate  individual 
differences  might  include: 

Display  format 

Information  requirements 

•  Type 

•  Aggregation 

Decision  naking/problem  solving  strategy 
Level  of  explanation 
Task  allocation 

The  adaptive  aid/interface  will  be  a  mixture  of  adaptive  and  prescriptive 
features.  Whether  the  feature  is  adaptive  or  prescriptive  is  determined  by 
performance.  A  prescriptive  feature  is  one  that  optimizes  the  performance  of 
all  users.  The  prescriptive  feature  is  preset  and  constrains  the  information 
and  decision  making  processes  the  user  can  use.  An  adaptive  feature  is  one 
that  can  be  changed  to  accommodate  user  characteristics  and/or  preferences,  and 
task  environment  characteristics.  The  form  of  the  adaptations  could  be  deter¬ 
mined  by  the  user  or  the  aid. 

Aiding  options  should  be  determined  by  the  option  that  optimizes  perform¬ 
ance.  If  different  options  are  optimal  for  different  users  and  these  optimal 
options  can  be  predicted  by  preferences  or  by  individual  differences,  then  the 
aid  should  be  adaptive  with  respect  to  the  option.  If  one  option  is  optimal 
for  all  users  then  this  option  or  characteristic  should  be  prescriptive.  How¬ 
ever,  the  mere  existence  of  individual  differences  in  users  does  not  necessi¬ 
tate  adaption  of  the  aid  to  those  differences.  Similarly,  if  a  user  prefers 
one  aid  feature  over  another,  the  aid  should  not  necessarily  accommodate  that 
preference.  Only  if  these  individual  differences  or  preferences  can  be  shown 
to  influence  performance  should  they  be  accommodated. 

It  is  likely  that  individual  differences  interact  with  task,  environment 
and  aid  features  to  determine  performance  and  acceptance.  That  is,  an  expert 
on  a  complex  task  using  summary  information  under  stress  might  perform  better 
than  a  novice  under  the  same  conditions  and  as  well  as  a  novice  on  a  simple 
task  using  detailed  information  under  no  stress. 
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Further,  the  different  individual  difference  variables  may  interact  (Huber, 
1981).  The  different  user  profiles  would  need  to  be  considered  with  each 
profile  requiring  a  different  aid  configuration.  Since  the  number  of  user 
profiles  increases  exponentially  with  the  number  of  relevant  individual  differ¬ 
ence  variables,  an  impossibly  complex  aiding  situation  would  be  created.  For 
this  reason,  Huber  is  pessimistic  about  individual  difference  research  leading 
to  design  guidelines.  However,  as  he  points  out,  the  magnitude  of  the  effect 
on  performance  of  many  of  the  individual  differences  is  likely  to  be  small, 
decreasing  significantly  the  number  of  individual  differences  that  need  to  be 
accommodated. 

Cognitive  style  and  expertise  are  variables  commonly  thought  to  be  impor¬ 
tant  to  the  design  of  information  systems  and  decision  support  systems  (e.g. , 
Bariff  &  Lusk,  1977;  Doktor  &  Hamilton,  1973;  Mason  &  Mitroff,  1973;  Sage, 

1981;  Zmud,  1979).  This  research  examined  the  effects  of  two  individual  dif¬ 
ferences,  cognitive  style  and  expertise,  on  patterns  of  information  usage.  The 
following  sections  will  present  a  brief  overview  of  the  literature  in  these 
areas. 

Expertise 

Expertise  is  an  individual  difference  variable  with  a  large  body  of  theory 
and  research.  These  theories  suggest  that  novices  and  experts  will  show  dif¬ 
ferent  patterns  of  information  usage  in  their  decision  making.  This  section 
briefly  reviews  current  theories  of  expertise.  While  specific  theories  of 
expertise  vary,  most  writers  in  this  area  agree  that  experts  have  a  great  deal 
more  knowledge  than  novices,  and  experts  have  more  experience  in  the  task  area. 
This  application  of  experience  to  knowledge  leads  to  different  knowledge  or¬ 
ganizations  and  different  mechanisms  for  accessing  the  knowledge.  These  in 
turn  lead  to  different  problem  solving  strategies  for  experts.  Each  of  these 
is  discussed  below. 

Experts  have  a  large  body  of  specialized  task  related  knowledge  (Lede, 

1986;  Feltovich,  1981,  1982;  Johnson,  Johnson,  &  Little,  1980;  Kolodner,  1984). 
This  knowledge  base  was  amassed  through  formal  education,  training,  and  a 
great  deal  of  relevant  experience.  This  experience  is  a  critical  difference 
between  the  novice  and  expert  (Feltovich,  1981;  Kolodner,  1984). 

Feltovich  (1981)  argues  that  not  only  are  the  content  and  size  of  the  ex¬ 
perts'  knowledge  bases  different  from  the  novices',  but  that  the  internal 
structuring  and  organization  of  the  knowledge  base  are  different.  Thus  he 
feels  expertise  is  a  matter  of  the  organization  of  knowledge  in  memory  and  the 
mechanism  for  accessing  the  appropriate  knowledge  when  needed.  In  his  review 
of  expert-novice  differences,  Feltovich  (1982)  found  that  novices  did  not  re¬ 
trieve  useful  knowledge,  or  they  retrieved  it  in  some  faulty  manner.  Feltovich 
suggests  that  experts  have  nultiple  classifications  of  the  same  event  in  a 
lattice  like  memory  organization.  Further,  they  have  learned  which  information 
is  most  useful  for  classification  and  problem  solving  and  which  is  tangential. 
Novices  attend  to  non-discriminating  cues.  Experts,  having  a  great  deal  of 
experience  in  their  field,  can  quickly  pick  out  the  features  relevant  to  the 
problem  and  establish  a  correspondence  between  these  features  and  an  internal 
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cognitive  model  of  the  problem.  This  means  they  classify  the  problems  much 
more  abstractly  than  novices.  All  of  this  suggests  that  the  novice  may  need 
help  in  discriminating  between  what  is  important  and  what  is  tangential  to  the 
problem. 

Feltovich  (1981)  speculates  that  many  kinds  of  logical  and  practical  group¬ 
ings  exist  for  the  expert,  tailored  to  different  problems,  contexts  and  differ¬ 
ent  phases  of  the  problem.  This  view  of  expert  problem  solving  is  consistent 
with  the  perceptual  chunking  theory  of  Chase  and  Simon  (1973).  According  to 
this  theory,  the  expert  has  chunked  together  the  characteristics  of  past  deci¬ 
sion  problems  with  the  actions  or  options  that  proved  effective.  Problem  solv¬ 
ing  for  the  expert  may  then  have  become  automated.  Rather  than  go  through  a 
series  of  formal  steps,  he  can  go  directly  from  the  problem  characteristics  to 
the  solution. 

Newell  (1973)  describes  experts  as  having  built  up  a  catalog  of  ready  made 
plans  based  on  a  great  deal  of  specific  knowledge.  Their  problem  solving  meth¬ 
ods  are  powerful  task  specific  procedures,  where  well  defined  conditions  must 
be  met  before  the  procedure  is  appropriate.  Experts  are  thus  able  to  pick  out 
the  most  appropriate  strategy.  Novices,  on  the  other  hand,  use  weak  general 
problem  procedures;  that  is,  they  use  formal  procedures  that  apply  widely  1  t 
have  a  low  success  rate. 

The  expert  relates  the  current  situation  to  patterns  previously  encounter¬ 
ed.  Only  when  the  problem  is  novel  does  the  expert  go  back  to  theoretical 
concepts  and  deductive  reasoning  (Larkin,  1981). 

A  similar  view  of  expert  problem  solving  is  proposed  by  Klein  (1981).  Ac¬ 
cording  to  Klein,  expert  performance  is  based  on  reasoning  by  analogy  rather 
than  by  analyzing  and  reintegrating  components,  as  novices  do.  Experts  do  not 
follow  explicit  rules  and  may  be  unable  to  describe  the  rules  they  do  follow. 
Expertise  is  grounded  in  the  ability  to  see  the  problem  in  terms  of  an  analo¬ 
gous  situation,  appreciation  of  the  significance  of  the  variables,  and  antici¬ 
pation  of  what  has  to  occur  to  achieve  a  goal.  Klein  presents  the  following 
model  of  proficient  decision  making. 

1.  The  decision  situation  is  perceived  in  terms  of  objectives. 

2.  Experience  suggests  analogous  situations  using  standard  operating 
procedures. 

3.  Similarities  and  differences  between  the  decision  situation  and  the 
analogous  situation  are  noted. 

4.  Analysis  of  the  analogous  situation  suggests  options  and  a  preferred 
option. 

5.  From  this  follows  adjustments  to  options  and  the  generation  of  new 
options. 
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Since  experts  tend  to  use  analogical  reasoning,  Klein  suggests  that  the 
decision  aid  should  support  the  recognitional  capacity  of  the  expert  to  help 
him  recognize  new  situations  in  terms  of  analogous  ones  and  use  them  to  define 
new  options.  The  novice,  on  the  other  hand,  might  be  best  supported  with  more 
formal  methods  since  he  cannot  use  previous  solutions  to  help  solve  the  prob¬ 
lem. 


Experts  in  different  subject  areas  use  different  problem  solving  strat¬ 
egies.  This  is  what  one  would  expect  if  expertise  were  in  part  a  matter  of 
knowledge  base  development.  Different  knowledge  base  organizations  support 
different  strategies.  Expertise  is  thus  task  specific  (Feltovich,  1982). 

The  difference  in  data  base  organization  and  decision  strategies  between 
novices  and  experts  means  that  the  two  groups  may  use  different  information. 
Experts  can  recognize  regularities  in  problem  fragments,  and  know  what  informa¬ 
tion  is  useful  for  problem  solving  and  what  is  irrelevant  (Feltovich,  1982). 

The  expert  learns  what  dimensions  are  important  to  the  decision,  and  he  is 
likely  to  consider  different  sets  of  information  than  the  novice  (Klein,  1981). 
Shanteau  (1985)  maintains  that  experts  have  highly  developed  perceptual  abili¬ 
ties.  They  are  able  to  extract  information  which  novices  cannot.  All  of  this 
suggests  that  novices  will  use  more  and  less  relevant  information  than  experts. 
This  conclusion  is  supported  by  Zmud  who  reviewing  the  literature  in  individual 
differences,  reports  that  subjects  with  a  great  deal  of  task  specific  knowl¬ 
edge,  engage  in  less  information  search. 

The  refined  problem  solving/decision  making  strategies  of  the  experts  lead 
to  other  characteristics:  experts  show  confidence  in  their  ability  to  make  de¬ 
cisions,  and  adaptability  and  responsiveness  to  changing  situations.  Experts 
have  a  high  stress  tolerance  which  may  derive  from  a  set  of  well  developed 
strategies  to  cope  with  common  problems,  but  often  they  cannot  articulate  the 
process  they  use  to  make  decisions  (Shanteau,  1985). 

The  latter  characteristic  can  be  explained  by  the  "chunking"  of  cognitive 
processes  (Chase  &  Simon,  1973)  that  may  accompany  experience  in  problem  solv¬ 
ing.  When  asked  to  describe  his  problem  solving  processes  the  expert  may  re¬ 
sort  to  reconstructing  what  he  thinks  they  should  be.  For  this  reason, 

Johnson,  Johnson,  &  Little  (1980)  suggest  that  the  study  of  expertise  should 
take  a  two  pronged  approach:  (1)  collect  records  of  performance  in  an  actual 
task  situation,  and  (2)  collect  records  of  their  accounts  of  their  performance. 
Johnson  et  al.  believe  the  matching  of  these  two  types  of  data  can  best  illumi¬ 
nate  the  processes  involved  in  expertise.  The  present  research  derived  and 
demonstrated  a  methodology  to  collect  records  of  decision  making  performance. 

One  purpose  of  this  experiment  was  to  examine  differences  in  the  informa¬ 
tion  use  of  two  groups  varying  in  experience  and  expertise.  The  above  review 
suggests  that  the  less  experienced  use  different  problem  solving  methods  than 
experts  and  require  different  types  of  information.  The  first  hypothesis  of 
this  experiment  was  that  the  less  experienced  will  require  a  greater  quantity 
of  and  more  specific  information  than  experts.  If  this  hypothesis  is  sup¬ 
ported,  it  will  suggest  that  inexperienced  users  may  need  different  kinds  of 
decision  support  than  experienced  users  and  that  aids  should  adapt  to  the 
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user's  level  of  expertise.  However,  the  latter  conclusion  is  contingent  on 
showing  that  a  user’s  performance  is  degraded  by  not  having  available  the  in¬ 
formation  appropriate  to  his  level  of  experience. 

Cognitive  Style 

Cognitive  style  is  defined  as  the  characteristic,  self  consistent  mode  of 
functioning  that  an  individual  shows  in  his  perceptual  and  intellectual  func¬ 
tioning  (Witkin,  Ortman,  Raskin,  &  Karp,  1971).  The  only  consensus  that  seems 
to  exist  in  the  literature  on  cognitive  style  is  that  it  is  a  multi-dimensional 
construct,  although  what  and  how  many  dimensions  are  involved  is  not  clear.  For 
example,  Bariff  (x  Lusk  (1977)  describe  three  cognitive  style  dimensions:  cogni¬ 
tive  complexity,  field  dependence/independence,  and  systematic/  heuristic. 
Huysmans  (1970)  defined  the  cognitive  style  dimension  of  "ways  of  reasoning" 
with  the  end  points  of  analytic  and  heuristic.  Mason  &  Mitroff  (1973)  used  the 
Jungian  typology  to  classify  cognitive  style  along  two  dimensions:  types  of 
information  acquisition  bounded  by  the  sensing  person  at  one  end  and  the  intui¬ 
tive  person  at  the  other,  and  types  of  information  processing,  bounded  by  the 
feeling  individual  and  the  thinking  individual.  Witkin  et  al.  (1971)  distin¬ 
guished  between  field  dependent  and  field  independent  perceptual  functioning 
with  corresponding  analytic  and  global  approaches  to  cognitive  functioning. 

Ragan  et  al.  (1978),  reviewing  the  cognitive  style  literature,  list  the 
following  ten  most  important  styles; 

1.  Field  dependent-independent 

2.  Impulsivity-ref lectivity 

3.  Visual-heptic 

4.  Leveling-sharpening 

5.  Constricted-flexible  control 

6.  Breadth  of  categorization 

7.  Scanning 

8.  Tolerance  for  unrealistic  experience 

9.  Cognitive  complexity 

10.  Conceptualizing  styles 

Cohen,  Bromage,  Chinnis,  Payne  and  Ulrila  (1982)  suggest  that  a  common 
thread  linking  many  of  the  cognitive  style  dimensions  in  the  literature  is  the 
distinction  between  analytic  style,  breaking  the  problem  down  into  elements, 
and  one  based  on  a  wholistic  approach  or  global  intuition.  In  any  case,  cogni¬ 
tive  style  is  not  a  well  defined  concept  composed  of  commonly  accepted  dimen¬ 
sions. 
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Extensive  research  demonstrates  that  people  differ  along  a  variety  of  cog¬ 
nitive  style  dimensions  (Bariff  &  Luck  (1977);  Huysman  (1970);  Mason  &  Hitroff 
(1973);  Ragan,  et  al.  (1978);  Witkin  et  al.  (1977)).  In  most  of  the  research, 
cognitive  style  was  measured  by  self-report  scales.  There  are  a  number  of 
studies  that  relate  cognitve  style  and  information  processing  or  decision  mak¬ 
ing  preference  or  behavior.  Henderson  &  Nutt,  (1980)  report  that  cognitive 
style,  as  measured  by  the  Myers  Briggs  Indicator  was  significantly  related  to 
assessment  of  risk.  Zmud  (1979),  reviewing  the  empirical  literature  on  indi¬ 
vidual  differences  and  the  success  of  management  information  systems,  reported 
contradictory  findings  with  respect  to  heuristic  and  systematic  cognitive  style 
types.  One  study  reports  that  systematics  prefer  more  information  while  an¬ 
other  reports  that  they  prefer  less.  Similarly,  Zmud  writes  that  systematics 
have  been  found  by  different  studies  to  prefer  both  aggregated  and  raw  data.  On 
the  other  hand,  the  literature  Zmud  reviewed  showed  consistent  findings  with 
respect  to  simple  and  complex  cognitive  styles.  Complex  subjects,  as  opposed 
to  simple  subjects,  prefer  to  search  for  and  use  more  information,  prefer  ag¬ 
gregate  to  raw  data,  use  more  complex  data,  generate  more  decision  alterna¬ 
tives,  have  less  confidence  in  their  judgment  and  use  more  time  to  make  the 
decision.  With  respect  to  field  dependent-independent  cognitive  style,  Zmud 
found  that  field-independents  prefer  detailed,  aggregate,  quantitative  reports 
and  require  more  decision  time.  Similarly,  Czarnolewski  (1987)  found  field 
dependence-independence,  as  measured  by  the  Embedded  Figures  Test  (Witkin,  et 
al.,  1971),  was  related  to  differences  in  information  processing. 

Another  question  to  address  is  whether  cognitive  style  is  related  to  per¬ 
formance,  e.g.,  the  quality  of  the  decision.  Despite  the  interest  in  and  pro¬ 
liferation  of  research  and  measures  of  cognitive  style,  there  is  little 
evidence  that  cognitive  style  is  related  to  differences  in  quality  of  decision 
or  problem  solving  success  (Moderick,  Levit,  Alden,  &  Henke  1975;  Sage,  1981). 
Moderick,  et  al.  suggest  that  the  effects  of  cognitive  style  should  be  found  in 
differences  in  the  processes  used  in  decision  making  and  not  in  terminal  per¬ 
formance  measures.  People  can  use  different  processes  and  produce  the  same 
decision  results.  Chervany  &  Dickson  (1978),  maintain  that  it  has  not 
been  possible  to  predict  performance  on  the  basis  of  personality  characteris¬ 
tics  (including  cognitive  style). 

Sage  (1981)  supports  this  conclusion,  and  presents  evidence  supporting  a 
cognitive  style  theory  that  incorporates  task  characteristics  and  the  decision 
maker's  task  experience.  According  to  Sage's  theory,  information  analysis  can 
be  done  in  a  concrete  operational  mode  or  a  formal  mode.  The  former  is  used  in 
familiar,  well  structured  solutions  and  involves  reasoning  by  analogy.  The 
formal  thought  process  uses  analytic  thought  and  is  applied  in  situations  with 
which  the  decision  maker  is  unfamiliar.  Sage  believes  that  it  is  the  decision 
makers'  experience  with  the  task  that  primarily  determines  the  approach  to 
information  acquisition  and  processing  and  decision  making.  He  argues  that  few 
processing  characteristics  are  invariant  over  the  decision  makers  and  tasks, 
but  that  the  characteristics,  or  styles,  evolve  with  experience.  This  suggests 
that  the  experience/expertise  dimension  will  be  related  to  information  proc¬ 
essing  differences  but  cognitive  style  per  se  will  not  be  related  to  differ¬ 
ences  in  information  processing  and  usage. 
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Sage's  theory  is  not  supported  by  the  data  reported  by  Czarnolewski  and 
Zmud,  cited  above.  However,  these  data  were  obtained  mainly  in  a  laboratory 
settings  with  college  students  using  unfamiliar  tasks.  The  present  experiment 
investigated  the  relationship  between  cognitive  style  and  information  usage  in 
an  applied  setting.  More  specifically,  it  was  hypothesized  that  cognitive 
style  is  related  to  pattern  of  information  usage  in  a  tactical  decision 
making  task. 
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METHODOLOGY 


To  investigate  the  use  of  battlefield  information  in  a  tactical  decision¬ 
making  problem  with  some  degree  of  control  and  recording  capability,  a  labora¬ 
tory  simulation  was  required.  The  design  and  operation  of  this  simulation  is 
described  below. 

Experimental  Task 

The  experimental  situation  involved  single  participants  acting  as  a  G3  (Op¬ 
erations)  Plans  Officer  of  an  Army  division.  The  G3  Plans  Officer  is  responsi¬ 
ble  for  planning  future  military  missions  based  upon  guidance  provided  by  the 
division  commander  and  the  Operations  (G3)  Officer.  The  G3  Plans  Officer  heads 
a  cell  of  from  20  to  27  officers  and  enlisted  personnel  which  includes  repre¬ 
sentatives  of  major  functional  areas  contained  in  the  division:  operations, 
intelligence,  logistics,  fire  support,  air  defense,  electronic  warfare,  nuc¬ 
lear/chemical  warfare,  and  engineers.  In  the  experiment  the  participant  was 
acting  alone.  The  information  in  the  data  base  was  to  substitute  for  the 
"normal"  inputs  from  the  various  functional  area  representatives. 

The  task  to  be  performed  was  to  define  a  recommended  concept  of  operations 
for  a  mission  to  be  undertaken  in  the  short  range  (i.e.,  within  the  next  12 
hours).  The  concept  was  to  include  an  operations  estimate,  a  recommended  task 
organization  for  the  mission  and  an  operations  overlay  that  graphically  por¬ 
trays  the  concept.  The  concept  of  operations  is  a  general  statement  of  how  a 
mission  is  to  be  accomplished;  the  general  objectives,  organization,  tasks  and 
timing  for  the  division  as  a  whole.  It  is  not  a  full-blown  plan,  but  rather 
provides  the  guidance  from  which  the  division  staff  produces  detailed  plans. 

The  operations  estimate  is  used  to  analyze  and  compare  the  possible  courses  of 
action  to  arrive  at  a  single  recommended  concept  of  operation  which  is  to  be 
presented  to  the  division  commander  for  his  decision. 

The  Information  upon  which  the  operations  estimate  is  based  is  drawn  in 
part  from  the  staff  estimates  produced  by  the  personnel,  intelligence  and  lo¬ 
gistics  staff  sections  in  which  they  analyze  the  possible  courses  of  action 
from  their  standpoints  to  arrive  at  a  recommended  one.  The  possible  courses  of 
action  analyzed  in  the  staff  estimates  come  typically  from  the  division  com¬ 
mander's  guidance  in  which  he  lists  the  courses  of  action  he  sees  as  available 
to  accomplish  the  mission  assigned  to  the  division  by  the  corps  commander. 
Figure  1  shows  the  military  decision  making  process  and  highlights  that  section 
of  the  process  accomplished  in  the  experimental  sessions. 

Experimental  Data  Base 

Two  original  scenarios  were  produced  and  tested  for  this  experiment.  These 
were  an  offensive  and  a  defensive  scenario  both  taking  place  in  Germany. 
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♦Process  covered  by  the  experimental  task.  Although  it  is  shown  as  a  command  process,  this 
estimate  can  be  produced  by  the  operations  (G3)  staff,  resulting  in  a  recommended  concept, 
rather  than  a  decision. 


1 

Figure  I.  The  Military'  Decisionmaking  Process 


1 


Taken  from  Reference  Book  RB  101-5,  "Command  and  Control  of  Operations," 
U.S.  Army  Command  and  General  Staff  College,  Fort  Leavenworth,  Kansas, 
June  1980. 
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It  was  decided  that  the  missions  for  the  experimental  task  would  occur  well 
into  the  battle  rather  than  at  the  outset  of  hostilities.  This  permitted  analy¬ 
sis  of  the  use  of  historical  as  well  as  current  information  in  the  development 
of  the  concept.  Thus,  the  battlefield  information  was  developed  for  the  cur¬ 
rent  situation  plus  three  historical  snapshots  typically  24  hours  apart. 

The  information  contained  in  the  data  base  consisted  of  data  that  might 
logically  be  available  to  the  G3  or  G3  Plans  Officer  at  the  time  he  is  required 
to  make  his  operations  estimate.  The  format  and  contents  of  the  reports  and 
overlays  comprising  the  data  base  were  derived  from  various  official  Army 
sources,  primarily  Field  Manual  101-5,  "Staff  Organization  and  Operations." 

The  textual/tabular  report  data  were  divided  into  the  four  primary  func¬ 
tional  areas  of  Personnel,  Intelligence,  Operations  and  Logistics.  Within 
these  functional  areas  the  separate  reports  comprised  data  categories.  Thus, 
within  the  Personnel  function  areas  were  the  data  categories  of  Personnel  Esti¬ 
mate,  Personnel  Strengths,  Losses  and  Gains  and  Other  Personnel  (i.e.,  prisoner 
of  war  data).  Each  of  the  data  categories  was  further  divided  into  data  ele¬ 
ments  consisting  of  the  separate  paragraphs  of  the  various  staff  estimates  and 
of  the  Corps  Operations  Order,  (i.e.,  the  detailed  order  received  from  higher 
headquarters)  or  of  the  separate  units  covered  within  the  various  status  re¬ 
ports.  For  instance,  the  Personnel  Strengths  data  category  consisted  of  seven 
data  elements,  one  for  each  of  the  units  for  which  personnel  strength  data  was 
available:  the  division  as  a  whole,  and  one  for  each  of  its  major  subordinate 

commands  (i.e.,  first  brigade,  second  brigade,  third  brigade,  the  division 
artillery,  the  division  support  command,  and  division  troops).  A  list  of  the 
data  categories  and  their  elements  is  contained  in  Appendix  B. 

For  purposes  of  analysis,  the  textual/tabular  data  elements  were  catego¬ 
rized  as  to  their  level  of  detail:  summarized,  aggregated  or  detailed.  Summa¬ 
rized  data  elements  were  those  that  contain  conclusions  and  inferences  such  as 
the  data  elements  in  the  various  staff  estimates  and  in  the  Corps  Operations 
Order.  Aggregated  data  elements  were  those  status  reports  which  had  been 
aggregated  for  the  division  as  a  whole,  typically  showing  status  data  only  down 
to  brigade  level.  In  our  example  the  division  data  element  in  the  Personnel 
Strengths  data  category  was  an  aggregated  element.  Detailed  data  elements  were 
reports  showing  status  data  down  to  at  least  battalion  level,  (i.e.,  one  eche¬ 
lon  below  brigade).  For  example,  all  of  the  other  data  elements  under  Person¬ 
nel  Strengths  were  at  the  detailed  level.  This  categorization  permitted 
analysis  of  data  use  in  terms  of  level  of  detail.  The  level  of  detail  assigned 
to  each  data  element  is  shown  in  Appendix  B. 

The  graphics  overlay  data  consisted  of  the  location  of  friendly  units  and, 
where  known,  enemy  units  down  to  maneuver  battalion  level.  Certain  support 
units  were  displayed  down  to  company/ battery  level.  For  maneuver  units,  the 
boundary  lines  and  assembly  areas  were  also  displayed  as  well  as  the  forward 
edge  of  the  battle  area  (FEBA)  trace,  and,  for  the  friendly  force,  the  fire 
support  coordination  line  (FSCL).  In  the  defense  scenario,  the  likely  enemy 
avenues  of  approach  were  available  in  standard  symbology.  In  the  offense  sce¬ 
nario  the  corps  operations  overlay  was  available  including  the  corps  objectives 


and  the  possible  axes  of  advance.  This  overlay  data  used  standard  military 
symbology.  Separate  overlays  were  made  for  the  current  situation  and  for  the 
three  historical  snapshots. 


To  summarize,  data  bases  were  developed  for  two  original  battle  scenarios. 
The  data  available  for  subject  viewing  consisted  of  typical  reports  and 
graphics  overlays  available  to  an  operations  officer  when  developing  his  opera¬ 
tions  estimate.  The  data  were  available  for  the  current  situation  and  three 
historical  snapshots  and  categorized  as  to  level  of  detail  for  analysis  pur¬ 
poses. 


Experimental  System  Design 


The  scenario  data  were  loaded  on  a  computer  system  for  presentation  to  the 
participant.  This  allowed  maximum  control  and  recording  capability  with  mini¬ 
mum  effort  during  the  experimental  sessions. 


The  computer  system  used  was  a  VAX  11/750  with  a  DeAnza  VC23  graphics  dis¬ 
play  generator.  Five  display  terminals  were  located  in  an  adjacent  room  and 
consisted  of  four  DEC  VTlOO-series  monochrome  terminals  with  keyboards  and  a 
single  DeAnza  19"  color  monitor  controlled  by  a  graphics  tablet. 


The  workstation  layout  is  shown  in  Figure  2.  The  System  Terminal  was  used 
to  set  experimental  parameters,  to  control  start/stop  times  and  to  initiate 
data  reduction.  The  Reports  Terminal  was  slaved  to  the  Interactor  Terminal.  It 
is  on  the  Reports  Terminal  that  the  participant  viewed  the  textual  and  tabular 
reports  concerning  the  battlefield  situation.  The  Planning  terminal  was  used 
as  an  "electronic  note  pad."  The  participant  requested  that  information  being 
displayed  on  the  Reports  Terminal  be  transferred  to  the  Planning  Terminal  or  he 
could  dictate  entry  of  notes  free  hand  on  the  Planning  Terminal  based  upon 
what  he  was  viewing  on  the  Reports  and  Graphics  terminals.  The  Interactor  Ter¬ 
minal  was  used  by  the  experimenter  who  performed  all  the  system  interactions. 

He  used  this  terminal  to  call  up  reports  which  the  participant  wished  to  see  on 
his  Reports  Terminal  and  to  initiate  transfer  of  data  from  the  Reports  Terminal 
to  the  Planning  Terminal. 


The  Graphics  Terminal  displayed  a  digitized  map  of  the  battlefield  area 
with  graphics  overlays  of  the  tactical  situation.  This  display  contained  simi¬ 
lar  information  to  the  printed  situation  maps  with  acetate  overlays  typically 
used  in  Army  command  posts.  Display  of  information  on  the  graphics  terminal 
was  controled  by  the  interactor  graphics  tablet.  The  participant  decided  the 
types  of  units  he  wanted  to  view.  The  types  of  terrain  data  to  be  displayed 
were  also  controlled  from  the  graphics  tablet  and  the  participant  could  request 
panning  around  the  battlefield  at  various  levels  of  resolution.  The  graphics 
capability  is  described  in  Appendix  C. 

_ 

The  listing  of  the  types  of  equipment  used  is  included  only  to  explain  the 
experimental  configuration.  It  does  not  in  any  way  represent  an  endorsement  by 
the  government. 


Three  products  were  required  of  the  participant:  the  Operations  Estimate, 
the  Operations  Overlay  and  the  Division  Task.  Organization  for  the  mission.  The 
Operations  Estimate  was  composed  on  the  Reports  Terminal  where  the  estimate 
outline  was  displayed  upon  request.  Thus  the  textual/tabular  reports  data  were 
not  available  for  viewing  during  estimate  composition.  This  forced  the  par¬ 
ticipant  to  extract  pertinent  data  onto  the  Planning  Terminal  allowing  the 
tracking  of  information  actually  used  in  composing  the  Operations  Estimate. 

The  Operations  Overlay  was  drawn  on  the  Graphics  Terminal  by  modifying  a 
copy  of  the  current  situation  overlay  by  means  of  moving  units,  changing  bound¬ 
aries  and  adding  other  control  measures  to  portray  the  concept  for  the  new 
mission.  The  Task  Organization  was  developed  on  the  Planning  Terminal  where  a 
split  screen  was  used  to  move  units  among  the  division  organizations. 

All  sessions  were  also  recorded  using  a  video  camera  and  wireless  micro¬ 
phones  on  the  participant  and  the  interactor.  The  use  of  the  system  is  des¬ 
cribed  in  more  detail  in  the  section  on  conduct  of  the  experiment. 

Participants 

Participants  for  the  experiment  were  drawn  from  two  sources.  The  first 
source  consisted  of  volunteers  from  the  Command  and  General  Staff  College 
(CGSC)  regular  course  at  Fort  Leavenworth.  This  is  a  ten-month  course  in  which 
mid-career  officers,  typically  majors,  are  taught  staff  operations  including 
military  decision  making  at  the  corps  and  division  levels.  Students  with  com¬ 
bat  arms  specialties  who  had  held  staff  positions  at  battalion  level  or  above 
were  requested.  Nine  such  students  volunteered  and  eight  of  them  were  used  in 
the  experiment  while  the  ninth  was  used  in  pilot  testing.  All  but  one  of  these 
were  from  combat  arms  specialties,  the  other  being  intelligence,  and  all  had 
had  some  staff  experience  at  battalion  or  above. 

The  second  source  of  participants  was  instructors  in  the  Combined  Arms 
Staff  and  Service  School  (CAS"^).  These  instructors  were  Lieutenant  Colonels. 

We  requested  that  they  also  be  from  combat  arms  specialties  and  have  held  staff 
positions  at  battalion  level  and  above.  Nine  subjects  were  designated  from 
among  the  instructors,  but  because  of  scheduling  problems,  not  all  of  these 
were  combat  arms  officers.  One  was  personnel,  one  was  engineer,  and  another 
intelligence;  the  remaining  six  were  from  combat  arms  specialties  and  one  had 
actually  been  a  G3  Plans  Officer  at  a  division.  Again,  eight  of  them  were  used 
in  the  experiment  and  the  other  one  for  pilot  testing. 

By  using  subjects  from  two  experience  levels,  we  hoped  to  determine  the 
effects  of  experience  on  patterns  of  information  use. 

All  16  participants  (eight  instructors  and  eight  students)  were  given  both 
the  offensive  and  defensive  problems  in  two  sessions,  separated  by  1-2  week  in¬ 
tervals.  The  order  of  presentation  was  counterbalanced  within  the  two  partici¬ 
pant  groups  to  counteract  any  learning  effects  on  the  scenario  conditions. 
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During  the  data  collection  sessions  with  the  first  set  of  four  instructors 
and  four  student  participants,  computer  system  problems  were  experienced.  These 
were  of  sufficient  magnitude  to  disrupt  almost  all  of  the  16  sessions  to  some 
degree.  Also,  some  scenario  and  procedural  problems  were  discovered  that  had 
not  been  apparent  during  pilot  testing.  All  of  these  problems  were  corrected 
and  the  data  collection  sessions  on  the  second  set  of  four  instructors  and  four 
students  ran  smoothly.  Thus  it  was  decided  to  ignore  the  sporadic  and  ques¬ 
tionable  results  from  the  first  set  of  participants  and  base  our  analysis  only 
upon  the  eight  participants  and  16  sessions  of  the  second  set.  As  it  turned 
out,  all  of  the  instructors  in  this  set  were  combat  arms  specialists  which 
provided  a  better  subject  comparison  than  the  set  of  instructors  taken  as  a 
whole. 

Conduct  of  the  Experiment 

Each  participant  received  an  hour  of  training  prior  to  his  first  experimen¬ 
tal  session.  This  hour  was  used  to  brief  the  participants  on  the  purpose  and 
procedures  of  the  experiment  and  to  familiarize  them  with  all  the  types  of  data 
available  in  the  system  and  how  the  data  were  presented.  This  was  done  at  the 
computer  workstation  using  a  training  scenario  that  stepped  them  through  the 
data  base  and  the  graphics  features  as  well  as  the  required  solution  products. 

The  first  thing  done  during  the  actual  experimental  sessions  was  to  intro¬ 
duce  the  participant  to  the  specific  offensive  or  defensive  situation.  This 
was  done  away  from  the  workstation  using  a  set  of  two  acetate  overlays  on  a 
1:250,000  scale  printed  map  of  the  area  of  interest.  A  standard  briefing  was 
used  which  first  explained  the  historical  buildup  to  the  current  situation 
using  the  first  overlay.  The  second  overlay  was  used  to  briefly  describe  the 
current  situation.  This  prebriefing  took  about  15  minutes. 

After  the  prebriefing,  the  participant  took  his  place  at  the  computer  work¬ 
station  and  began  requesting  the  display  of  data  he  considered  necessary  to 
solve  the  problem.  He  was  entirely  free  to  view  whatever  data  he  desired  in 
whatever  order  he  desired  to  view  it.  The  only  restriction  was  a  time  limit  of 
four  hours  to  complete  the  problem.  All  system  interactions  were  performed  by 
an  "interactor"  who  responded  to  the  participant’s  verbal  requests  for  informa¬ 
tion  by  calling  up  the  requested  data. 

The  participant  also  took  notes  on  his  "electronic  note  pad"  (i.e.,  the 
Planning  Terminal)  either  by  requesting  the  transfer  of  lines  of  data  from  the 
Reports  Terminal  to  the  Planning  Terminal  or  by  dictating  "free  hand"  notes  to 
the  interactor  based  upon  things  he  was  viewing.  Participants  organized  their 
notes  by  assigning  titles  to  an  empty  outline  on  the  Planning  Terminal.  For 
example,  a  participant  might  title  a  note  section  "Enemy  Situation"  and  enter 
all  notes  concerning  the  enemy  under  that  section.  Some  participants  outlined 
their  note  pads  as  the  first  step  in  a  session,  others  did  it  as  they  went 
along. 
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Participants  were  free  to  do  the  three  parts  of  the  solution  whenever  and 
in  whatever  order  they  desired.  Again,  the  interactor  did  all  the  recording 
based  upon  the  participant's  instructions.  The  only  exception  to  this  was  that 
several  participants  typed  their  Operations  Estimates  themselves,  using  the 
Interactor's  Terminal. 

At  the  conclusion  of  each  session,  participants  were  asked  to  complete  a 
two  part  questionnaire.  The  first  part  asked  participants  to  indicate  their 
level  of  agreement,  on  five  point  scales,  with  statements  about  the  adequacy  of 
the  training  session,  the  information  base,  graphics,  experimental  procedures, 
and  scenarios.  The  second  part  consisted  of  three  open  ended  questions  asking 
about  the  information  base  and  the  value  of  the  procedure  in  extracting  useful 
data  on  tactical  decision  making.  Appendix  D  contains  a  copy  of  the  question¬ 
naire. 

After  the  first  experimental  session  for  each  participant,  they  were  given 
the  Embedded  Figures  Test  (EFT).  The  EFT  is  a  measure  of  cognitive  style  that 
requires  the  locating  of  previously  viewed  geometric  figures  embedded  in  larger 
complex  figures.  The  underlying  assumption  is  that  perceptual  performance  is 
related  to  cognitive  performance.  Thus,  the  ability  to  overcome  an  embedding 
context  is  believed  to  be  related  to  a  global-analytic  dimension  of  cognitive 
functioning.  The  global  person  tends  to  view  the  total  undifferentiated  situa¬ 
tion,  while  the  analytic  person  breaks  up  the  whole  into  parts,  reorganizing 
and  combining  them.  It  was  hypothesized  that  the  global  person,  as  differenti¬ 
ated  by  the  EFT,  would  in  our  experiment  use  less  information  and  prefer  less 
detailed  information.  The  analytic  person  would  use  more  information  and  use 
both  summarized  and  detailed  data. 

The  EFT  was  administered  at  the  end  of  the  first  experimental  session 
because  pilot  subjects  tended  to  perceive  the  EFT  as  an  aptitude  test  and,  de¬ 
spite  our  reassurances,  to  show  resistance  to  taking  the  test.  The  experimen¬ 
tal  session,  on  the  other  hand,  involved  material  familiar  to  the  subject  and 
an  environment  in  which  he  was  comfortable.  After  participating  in  the  first 
experimental  session,  subjects  were  more  relaxed  and  showed  no  anxiety  over 
taking  the  EFT. 

We  selected  the  EFT  instead  of  other  measures  of  cognitive  style  because  of 
its  validity  and  reliability  and  the  relatively  short  time  required  to  admini¬ 
ster.  Also,  the  EFT  is  not  susceptible  to  social  desirability  bias  as  is  true 
of  most  of  the  other,  more  direct,  measures  of  cognitive  style.  A  more  com¬ 
plete  description  of  the  EFT  and  the  reasons  for  its  selection  as  a  measure  of 
cognitive  style  is  contained  in  Appendix  A. 

Data  Analysis 

All  system  interactions  were  recorded  automatically  in  a  data  file.  This 
included  actions  against  both  the  textual/tabular  data  and  the  graphics  data. 
Information  recorded  included  the  time;  type  of  action;  the  functional  area, 
data  category,  and  data  element  involved  in  the  transaction;  and  the  age  and 
level  of  detail  of  the  data  involved.  If  the  action  involved  data  entry,  the 
contents  of  the  entry  also  were  saved. 
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Typically,  it  is  a  long  and  arduous  task,  to  analyze  process  data  involving 
the  viewing  and  reduction  of  hours  of  video  and  audio  recordings.  Had  data 
reduction  software  been  available  (and  its  development  is  quite  feasible),  we 
could  have  accomplished  in  a  matter  of  hours  what  took  months  to  complete. 

As  it  were,  we  had  printouts  of  chronological  event  listings  for  each 
session  which  were  also  listed  by  terminal.  Printouts  of  the  contents  of  the 
working  files  and  completed  operations  estimates  also  were  available.  The  only 
data  reduction  program  available  summarized  time  usage  by  terminal. 

The  variables  of  interest  were  the  two  participant  groups  (students  vs.  in¬ 
structors),  the  two  scenarios  and  the  within  subject  differences.  Comparisons 
within  these  variables  were  made  on  a  variety  of  time,  sequence,  content,  and 
product  measures  reported  in  the  following  section. 

A  more  thorough  description  of  the  laboratory  system  is  contained  in  "Com¬ 
mand  and  Control  (C^)  Laboratory  Concept  Evaluation,  Final  Report,"  a 
multi-volume  working  paper  covering  the  laboratory  development 
(McKeown,  et.al.,  1985). 
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RESULTS 


Table  1  summarizes  the  findings  of  this  exploratory  research.  The  general 
summation  is  that  large  variances  occurred  between  the  sessions  which  are  not 
accounted  for  by  any  of  the  four  independent  variables:  student  vs  instructor 
(experience),  offense  vs.  defense  scenario,  first  session  vs.  second  session  or 
Embedded  Figures  Test  scores. 

The  student/instructor  comparison  did  show  two  strong  differences:  stu¬ 
dents  looked  at  more  data  than  did  instructors  during  the  decision  making  proc¬ 
ess  and  a  greater  percentage  of  the  data  viewed  by  students  was  detailed  in 
nature  while  the  instructors  viewed  a  greater  percentage  of  summary  type  infor¬ 
mation.  This  was  reflected  in  the  instructors'  concentration  on  the  summary 
information  in  the  staff  estimates,  corps  operations  order  and  division  com¬ 
mander's  guidance. 

Although  there  were  no  quantifiable  measures  of  the  problem  solution  in 
this  research,  an  overall  view  of  the  solutions  suggest  that  the  instructors 
tend  to  be  more  conservative.  The  way  they  positioned  their  reserves  and  employed 
units  indicates  a  greater  concern  for  meeting  possible  contingencies.  Stated 
another  way,  it  appears  that  the  instructors  may  have  been  more  adverse  to  risk 
than  were  the  students. 

Beyond  these  points,  no  differences  were  found  between  the  students  and  the 
instructors.  There  were  large  differences  across  sessions  in  the  specific  data 
used  and  the  data  that  were  considered  important  enough  to  note,  but  these  were 
not  related  to  whether  the  participant  was  a  student  or  an  instructor.  Indeed, 
there  were  wide  differences  even  in  the  data  individual  participants  used  be¬ 
tween  their  first  and  second  sessions. 

This  data  utilization  difference,  however,  was  not  related  to  the  type  of 
mission,  (i.e.,  offensive  versus  defensive)  the  participant  is  facing.  None  of 
the  measures  taken  showed  any  effect  of  mission  type.  The  effect  of  session 
position  (i.e.,  first  session  vs.  second  session)  was  a  relatively  weak  tend- 
ancy  (p  <  .05)  for  the  second  session  to  take  less  time  to  complete  than  the 
first.  This  suggests  that  some  training  occurred  despite  the  attempts  to 
moderate  this  confounding  variable. 

Even  with  the  individual  differences,  there  was  a  clear  pattern  of  data  use 
across  all  participants  when  viewed  at  the  macro  level  of  functional  area  use. 

The  typical  participant  first  looked  at  the  mission  requirements  and  command¬ 
er's  guidance  plus  the  status  of  his  own  forces.  He  then  studied  the  terrain 
and  the  enemy  forces.  He  then  went  back  to  look  at  supporting  data,  taking 
them  in  order,  beginning  with  personnel.  Finally,  he  returned  to  the  opera¬ 
tions  and/or  intelligence  data  to  confirm  information  important  to  the  concept 
he  was  developing. 

The  Embedded  Figures  Test  (EFT)  scores  were  not  predictive  of  participant 
performance  on  this  task.  There  was  considerable  variance  among  the  partici¬ 
pant  EFT  scores  but  these  did  not  correlate  with  the  number  of  data  elements 
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viewed  nor  the  level  of  detail  of  the  information  used.  Only  total  data  search 
time  correlated  significantly  with  EFT  scores  although  seven  hypothetically 
related  measures  were  compared. 

The  participant  questionnaire  results  generally  supported  the  use  of  the 
research  method  for  tactical  decision  making  research.  Many  qualifications 
were  given  however,  and  suggestions  made  for  improvements.  A  strong  finding 
from  both  the  questionnaire  data  and  comments  made  during  the  session  was  a 
lack  of  satisfaction  with  the  computer  graphics.  The  most  common  complaint  was 
difficulty  in  maintaining  geographical  orientation  when  panning  around  the 
division  area  of  interest  during  terrain  and  force  analysis.  If  participants 
used  a  large  enough  scale  to  see  the  entire  division  area  at  once,  then  clutter 
and  lack  of  resolution  became  a  problem.  All  of  them  would  have  preferred  to 
use  standard  wall  maps  with  overlays. 

A  detailed  description  of  the  findings  for  each  of  the  categories  of  meas¬ 
ures  taken  is  contained  in  Appendix  D. 
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DISCUSSION 


Individual  Differences 


All  of  the  participants  were  military  professionals  and  very  familiar  with 
the  military  decision  making  process.  Thus,  the  approaches  they  used  to  solve 
this  problem  were  rational  and  much  more  homogeneous  than  would  have  been 
expected  had  naive  participants  been  used.  For  instance,  there  was  little 
evidence  of  compulsive  data  use;  the  participants  were  able  to  distinguish 
between  what  information  is,  or  may  be,  pertinent  to  the  problem  from  what  is 
irrelevant.  This  is  evidenced  in  such  things  as  the  ignoring  of  prisoner  of  war 
data  and  the  limited  use  of  Class  III  and  Class  V  status  data  when  no  problems 
existed  in  those  areas.  Also  none  of  the  participants  seemed  particularly 
rushed  by  the  four  hour  time  limit,  but  allotted  their  time  wisely.  This  makes 
the  differences  that  were  observed  all  the  more  typical  of  actual  tactical 
decisionmaking. 

There  were  large  individual  differences  in  how  this  homogeneous  group  ap¬ 
proached  the  problem  and  the  information  items  they  used  to  reach  a  decision, 
yet  these  differences  were  not  generally  related  to  the  type  of  scenario  nor  to 
the  f irst-versus-second  session  variable.  The  individual  differences  also  were 
not  predictable  using  our  measure  of  cognitive  style,  the  Embedded  Figures  Test 
(EFT).  Perhaps  most  surprising,  given  the  lack  of  scenario  and  order  effects, 
was  the  lack  of  consistency  within  individuals  in  the  types  of  data  items  they 
used  in  analyzing  the  problem. 

Perhaps  the  most  likely  cause  for  this  finding  would  be  that  our  eight 
participants  generally  did  not  approach  this  problem  with  a  definite  schema  in 
mind  for  solving  it.  There  was  a  high  degree  of  interest  in  the  task  but  also 
there  was  evidence  of  some  experimentation  going  on  as  suggested  by  participant 
statements  like,  "I  know  I'm  suppose  to  do  it  X  way,  but  1  want  to  try  Y."  The 
session  atmosphere  was  generally  relaxed  and  it  is  probable  that  the  partici¬ 
pants  used  it  as  an  opportunity  to  try  out  various  approaches. 

This  does  not  invalidate  the  findings.  Much  of  the  wargaming  and  command 
group  decision  making  process  is  exactly  the  sort  of  "what  if"  experimentation 
used  by  our  participants.  The  lack  of  standard  schemas  in  determining  a  course 
of  action  might  therefore  be  more  indicative  of  battlefield  decisionmaking  than 
had  we  found  a  "lock  step"  use  of  data.  Also,  even  though  there  was  much  vari¬ 
ance  in  how  the  solutions  were  derived  there  was  little  variance  in  the  general 
solutions  themselves.  This  suggests  that  if  the  data  base  elements  are  repre¬ 
sentative,  then  there  exist  many  possible  ways  to  arrive  at  the  same  general 
conclusion. 

For  example,  there  was  considerable  redundancy  between  the  information  con¬ 
veyed  by  unit  status  reports  and  that  conveyed  by  staff  estimates;  the  latter 
being  generally  a  summary  (with  conclusions)  of  the  former.  To  a  lesser  extent 
situation  graphics  and  alphanumeric  reports  conveyed  similar  information.  In 
one  session,  a  participant  attempted  to  use  graphic  information  almost  exclu¬ 
sively  in  developing  his  estimate  and  concept  of  operation. 
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This  redundancy  of  information  makes  it  possible  to  take  several  routes  to 
the  same  goal.  The  cautious,  detailed  individual  can  study  the  details  of  unit 
status  reports.  The  generalist  "big  picture"  type  can  ignore  all  detailed  data 
and  look  only  at  processed,  summarized  information.  The  spatially  oriented 
individual  can  concentrate  on  graphic  situation  displays.  However,  the  results 
of  this  study  indicate  that  the  professional  officer  may  not  be  readily  stereo¬ 
typed  as  a  particular  type  of  data  user.  He  tends  to  use  various  types  of 
information  as  his  thought  processes  guide  him  through  the  decision  making 
task. 

This  finding  has  considerable  implication  for  tactical  decision  support 
system  design.  First,  decision  aids  are  frequently  built  by  using  a  single 
expert  advisor  and  if  multiple  experts  are  used  they  are  typically  asked  to 
reach  a  consensus.  Our  results  suggest  chat  the  products  of  this  design  meth¬ 
odology  will  not  be  acceptable  in  a  population  with  such  variance  in  approach. 
Second,  the  variability  of  information  use  suggests  that  adaptability  is  an 
important  feature  for  tactical  decision  aids.  Required  adaptability  however 
may  be  limited  in  scope  as  suggested  by  the  strong  typical  pattern  in  overall 
approach.  This  typical  pattern,  as  indicated  in  Table  3,  suggests  a  general 
framework  for  decision  support  of  the  operations  estimate  task.  This  may  be 
true  of  other  operations  tasks  as  well  and  points  out  an  important  benefit  of 
this  type  of  data  collection. 

Also,  this  study  looked  at  the  relationship  between  individual  differences 
and  patterns  of  information  usage.  The  results  indicate  that  although  partici¬ 
pants  preferred  to  use  different  patterns  of  information,  use  of  different 
patterns  did  not  produce  different  task  solutions.  Before  concluding  that  an 
adaptive  aid  is  needed  to  accommodate  different  levels  of  experience,  the  pos¬ 
sibility  should  be  investigated  that  users,  despite  preferences  for  different 
patterns  of  information  usage,  can  easily  and  without  discomfort  or  degrading 
their  performance,  use  non-preferred  patterns  of  information. 

Results  did  not  show  a  statistically  significant  relationship  between  cog¬ 
nitive  style,  as  measured  by  the  EFT,  and  patterns  of  information  usage.  This 
finding  is  consistent  with  the  literature  relevant  to  the  person-situation 
debate  over  whether  behavior  is  situationaiiy  specific  oi  i^  determined  by 
broad  personality  traits.  Proponents  of  the  former  position  argue  that  al¬ 
though  correlations  are  generally  found  between  personality  traits  and  behav¬ 
ior,  personality-traits  account  for  less  than  lOJi  of  the  variance  in  the  behav¬ 
ioral  measures,  and  the  situation  accounts  for  about  3C%  of  the  variance. 
Epstein,  and  O'Brien  (1985)  argue  that  this  body  of  research  is  flawed  ^n  that 
measuring  a  single  instance  of  behavior  is  equivalent  to  using  a  one  item  test, 
which  makes  it  a  measure  of  questionable  reliability  and  validity.  Rather, 
behavior  measures  should  be  aggregated  over  occasion  and  situation.  When  this 
is  done,  Epstein  and  O'Brien  report  that  there  is  strong  evidence  for  stable 
broad  personality  traits.  Crumley  (personal  communication,  April  2,  1986) 
maintains  that  the  same  argument  applies  to  the  research  on  cognitive  styles. 
Measures  of  behavior,  to  be  reliable,  should  be  taken  over  several  situations 
and  tasks. 
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This  study  looked  at  one  instance  of  behavior  and  one  task;  the  Epstein  and 
O'Brien  argument  may  well  account  for  our  failure  to  find  information  usage 
patterns  attributable  to  cognitive  style.  However,  we  were  not  prepared  to 
investigate  the  existence  of  broad  stable  preferences  in  information  usage 
attributable  to  cognitive  style,  but  whether  the  influence  of  cognitive  style 
was  sufficiently  important  to  produce  a  measurable  effect  in  this  particular 
task. 

Amid  all  the  inter  and  intra  individual  variability  there  were,  however, 
two  significant  effects  of  group  membership:  the  instructors  viewed  signifi¬ 
cantly  fewer  data  elements  than  did  the  students  and  among  those  that  were 
viewed  a  larger  proportion  of  them  were  summary  element  and  a  smaller  propor¬ 
tion  of  them  were  detailed  elements  for  instructors  than  for  students.  This  is 
a  predictable  outcome  in  light  of  prior  research  discussed  earlier.  However, 
the  past  research  has  typically  compared  the  decision  making  behavior  of 
novices  and  experts,  concluding  that  novices  have  only  weak  general  decision 
making  procedures  to  work  from  whereas  experts  have  a  set  of  task  specific 
procedures  which  are  much  more  efficient  (Newell,  1973). 

But  none  of  our  participants  could  be  classified  as  novices  in  this  sense. 
The  students  all  had  the  rank  of  major  with  at  least  13  years  of  service.  All 
were  combat  arms  officers  with  some  staff  experience.  All  had  received  exten¬ 
sive  classroom  instruction  and  exercises  in  the  tactical  decision  making  proc¬ 
ess  during  the  preceeding  six  months.  Thus  the  difference  in  experience 
between  the  two  groups  is  a  relatively  minor  one  compared  to,  say,  captains 
versus  lieutenant  colonels  or  majors  versus  colonels. 

Andriole  (1984)  also  found  performance  differences  between  relatively  simi¬ 
lar  groups  of  Army  officers.  Andriole  used  two  groups  of  decision  makers  from 
the  Army  War  College,  three  lieutenant  colonel  students  in  one  group  and  three 
colonel  instructors  in  the  other.  Each  group  was  required  to  develop  a  concept 
of  operation  using  the  Letort  scenario  extant  at  the  Army  War  College.  The 
process  took  three  hours  for  each  group  and  the  sessions  were  recorded  for 
later  analysis.  From  his  analysis  Andriole  concluded  that: 

•  The  colonels  showed  considerably  more  risk  aversion  than  the  lieutenant 
colonels . 

•  The  colonels  generated  more  options  yet  were  less  confident  in  their 
solution. 

•  The  colonels  followed  doctrine  closer. 

•  The  lieutenant  colonels  were  less  apt  to  challenge  their  solution, 
little  evidence  of  "devil's  advocate"  play. 

Andriole  reports  no  direct  measures  of  data  use  so  no  comparisons  can  be 
made.  However,  it  is  interesting  that  he  found  the  more  senior  officers  to  be 
more  conservative  and  doctrinal  in  their  solution,  similar  to  our  findings. 
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The  implication  is  that  if  relatively  minor  differences  in  experience  can 
produce  significant  differences  in  the  type  and  amount  of  information  used  and 
in  the  way  that  information  is  interpreted  in  tactical  decision  making,  then  it 
is  important  to  base  decision  support  systems  upon  the  behavior  of  the  specific 
user  population.  For  instance,  if  a  decision  aid  is  being  developed  specific¬ 
ally  for  division  commanders,  then  data  upon  which  to  base  that  system  should 
be  collected  from  general  officers  who  have  commanded  divisions.  To  design 
such  an  aid  based  upon  the  opinions  and  behavior  of  subordinate  staff  officers 
may  be  misleading. 

Methodology 

Another  concern  of  this  study  was  to  test  the  effectiveness  of  the  method¬ 
ology  in  gathering  information  on  tactical  decision  making.  Although  the  par¬ 
ticipants  generally  agreed  that  the  method  would  be  effective,  several 
improvements  are  required  to  enhance  its  usefulness.  These  are  discussed  be¬ 
low  . 


1.  Provide  data  reduction  routines  to  manipulate  and  analyze  the  captured 
performance  data. 

Automated  systems  like  this  offer  tremendous  advantages  in  analyzing  proc¬ 
ess  data.  However  the  objective,  reliable  capturing  of  performance  parameters 
is  of  limited  value  if  one  must  still  go  through  the  tedious  process  of  reduc¬ 
ing  this  information  from  chronological  event  listings.  It  is  a  reasonably 
straightforward  process  to  develop  data  reduction  programs  for  the  system  that 
will  do  most  of  this  for  you  much  faster,  cheaper,  and  more  reliably.  As  most 
of  the  experiments  using  the  system  will  be  of  the  same  general  type  using  the 
same  or  similar  dependent  measures,  general  data  reduction  routines  can  be 
developed.  The  experimenter  interface  would  then  permit  the  selection  and 
interfacing  of  individual  reduction  routines  to  match  specific  experiment  re¬ 
quirements  . 

2.  Improve  the  performance  data  capture  of  the  system. 

This  is'  another  area  where  an  automated  laboratory  environment  has  tremen¬ 
dous  potential.  Ideally,  one  could  envision  a  laboratory  system  where  the 
effects  of  an  individual  item  of  information  on  a  decision  could  be  isolated 
and  measured,  thus  mapping  the  cognitive  processes  involved  in  a  much  more 
objective  manner  than  is  currently  possible.  The  objective  capture  of  process 
data  has  implications  for  modeling,  aids  development  and  evaluation,  and  deci¬ 
sion  making  training  and  feedback.  A  goal  of  the  permanent  command  and  control 
laboratory  facility  being  developed  at  Fort  Leavenworth  is  to  evolve  an  experi¬ 
ment  facility  that  maximizes  objective  performance  data  capture. 

3.  Provide  multiple  participant  workstations. 

One  of  the  sources  of  artificiality  in  our  data  collection  effort  was  the 
fact  that  the  experimental  task  was  performed  by  individuals  working  in  isola¬ 
tion.  In  actual  operations  the  G3  Plans  Officer  has  a  staff  with  which  he 
interacts  to  reach  decisions.  He  also  can  discuss  the  matter  with  his  G3  and 
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other  staffs.  In  fact,  few  decisions  made  in  a  command  post  are  arrived  at  by 
a  single  individual  acting  in  isolation.  What  is  required  are  multiple  work¬ 
stations  that  permit  team  efforts  and  data  sharing  while  still  allowing  objec¬ 
tive  data  capture.  Also  the  development  and  testing  of  intra  staff  automation 
communication  aids  are  necessitated  by  the  automation  and  dispersion  of  future 
military  field  command  posts.  The  permanent  laboratory  facility  at  Fort 
Leavenworth  as  described  in  Crumley,  (1985),  will  provide  this  capability. 

4.  Improve  the  man-machine  interface  to  facilitate  ease  of  participant 
operation. 

The  system  used  in  this  data  collection  required  more  time  than  was  availa¬ 
ble  to  Crain  participants  to  a  common  proficiency  in  its  operation.  For  this 
reason  a  skilled  interactor  was  used  to  perform  the  required  system  interac¬ 
tions.  A  more  "user  friendly"  interface  will  permit  participant  operation  in 
future  experiments. 

5.  Improve  and  expand  the  scenarios. 

There  were  several  complaints  about  various  items  that  were  not  included  in 
the  data  base.  Also  it  was  apparent  from  the  uniformity  of  Che  solutions  chat 
more  complexity  needs  to  be  added  to  the  scenarios  to  get  discrimination  in  the 
solutions.  Further,  additional  data  sets  must  be  added  to  make  the  scenarios 
useable  to  gather  performance  data  on  other  tasks  such  as  creation  of  a  divi¬ 
sion  operations  order  and  multiparticipant  interactions.  Efforts  are  underway 
to  expand  and  revise  these  scenarios. 

An  additional  finding  that  deserves  some  discussion  is  the  difficulty  the 
participants  experienced  in  using  the  computer  graphics.  The  most  common  com¬ 
plaint  with  using  the  system  was  that  it  was  difficult  to  obtain  a  comprehen¬ 
sive  understanding  of  the  situation  using  the  small  (19")  video  display. 
Participants  were  accustomed  to  using  large  (e.g.,  4'x8')  wall  maps  with  ace¬ 
tate  overlays  which  show  the  entire  area  of  interest  at  a  glance.  With  the 
video  displays,  if  they  went  to  a  scale  large  enough  to  display  the  entire  area 
of  interest,  the  clutter  of  symbols  was  too  great  to  read  and  the  terrain  reso¬ 
lution  was  too  low  to  be  of  any  analytical  value.  If  they  went  to  a  scale 
small  enough  to  avoid  these  problems  then  only  a  small  portion  of  the  divi¬ 
sion's  area  of  interest  could  be  displayed  at  one  time.  This  frequently  caused 
problems  with  orientation  concerning  where  units  and  terrain  features  not  cur¬ 
rently  being  displayed  were  in  relation  to  the  current  display. 

This  problem  would  be  helped  some  by  the  use  of  high  resolution  graphics. 
Also  the  use  of  windows  in  modern  graphics  systems  would  allow  a  pictorial  rep¬ 
resentation  of  where  the  current  display  is  in  relation  to  the  "big  picture." 
Other  techniques  might  be  tested  such  as  displaying  the  symbols  for  adjacent 
units  at  the  edges  of  the  display  with  an  indication  of  their  location  distance 
from  that  edge.  Although  these  techniques  will  probably  help,  the  nature  of 
command  post  operations  often  involve  interactive  group  decision  making  around 
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a  situation  map.  This  is  such  an  integral  part  of  tactical  decision  making 
that  it  may  be  that  only  high  resolution  large  screen  graphic  systems  will  be 
able  to  completely  replace  acetate  situation  maps  in  command  posts. 

Automation  of  command  post  operations  is  now  inevitable  and  workable  solu¬ 
tions  to  these  problems  must  be  achieved.  Also  there  are  many  advantages  of 
computer  graphics  such  as  wargaming,  data  updating,  overlay  generation  and  ter¬ 
rain  analysis.  These  advantages  and  others  need  to  be  enhanced  to  ease  the 
switch  over  to  automtion  and  motivate  user  solutions  to  the  organizational  and 
operational  impact  of  these  systems.  The  new  ARI  laboratory  facility  at  Fort 
Leavenworth  is  dedicated  to  investigating  these  problems  and  working  toward 
viable  solutions  for  the  automated  command  and  control  systems  of  the  future. 
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SUMMARY  AND  CONCLUSIONS 


Military  officers  acted  as  Army  division  operations  (G3)  plans  officers. 
They  were  asked  to  use  whatever  data  they  desired  from  a  computerized  data  base 
of  typical  command  post  alphanumeric  and  graphic  information  to  develop  an 
operations  estimate  and  an  operations  overlay  and  task  organization  for  their 
recommended  course  of  action.  Each  participant  was  given  two  different  scenar¬ 
ios.  The  computerized  system  was  designed  for  machine  capture  of  the  partici¬ 
pants'  data  use.  One  purpose  was  to  determine  if  patterns  of  data  use  could  be 
distinguished  and  how  they  relate  to  the  solution  and  also  to  test  the  data 
capture  methodology. 

There  were  large  individual  differences  in  data  use,  both  between  partici¬ 
pants  and  between  the  two  sessions  for  individuals.  These  differences  were  not 
related  to  the  type  of  scenario  nor  were  they  predicted  by  a  measure  of  cogni¬ 
tive  style  (i.e.,  Embedded  Figures  Test).  Lieutenant  colonel  instructors 
viewed  significantly  fewer  information  elements  and  a  larger  proportion  of  the 
elements  viewed  were  summary  items  than  did  majors  who  were  students  at  the 
Army  Command  and  General  Staff  College  (CGSC).  Solutions,  although  varying  in 
detail,  were  very  homogeneous  in  the  general  course  of  action  selected. 

From  these  findings  it  is  concluded  that  even  reasonably  homogeneous  groups 
of  tactical  decision  makers  will  vary  in  the  information  they  prefer  to  use  and 
how  they  use  it  in  reaching  a  decision.  It  remains  to  be  determined  if  requir¬ 
ing  individuals  with  diverse  decision  making  patterns  to  use  a  standard 
approach  will  adversely  affect  their  decision  making  abilities. 

Further,  the  findings  indicate  that  even  relatively  small  differences  in 
experience  levels  will  create  significant  differences  in  the  amount  and  type  of 
information  used  in  tactical  decision  making.  Also,  the  findings  suggest  that 
experience  may  be  a  better  predictor  of  information  usage  than  are  measures  of 
cognitive  style.  Thus,  experienced  target  user  representative  populations 
should  be  used  to  gather  information  for  decision  aid  development. 

Finally,  it  is  concluded  that  a  laboratory  setting  can  be  used  to  collect 
useful  information  on  command  and  control  task  behavior.  However,  the  labora¬ 
tory  system  must  be  well  instrumented  and  possess  reasonable  fidelity  in  terms 
of  staff  interactions  and  realistic  scenarios. 


REFERENCES 


Andriole,  S.J.  (1934),  The  design  and  development  of  an  intelligent  planning 
aid.  Marshall,  VA:  International  Information  Systems,  Inc. 

Bariff,  M.L.  &  Lusk,  E.J.  (1977).  Cognitive  and  personality  tests  for  the 

design  of  managment  information  systems.  Management  Science,  23 ,  820-829. 

Bauman,  G.  (1951).  The  stability  of  the  individual's  mode  of  perception  and 
of  perception-personality  relationships.  Unpublished  doctoral  disserta¬ 
tion,  New  York  University. 

Benbasat,  I.,  &  Taylor,  R.N.  (1978).  The  impact  of  cognitive  styles  on 
information  system  design.  MIS  Quarterly.  _2,  43-54. 

Chase,  W.C.,  &  Simon,  N.A.  (September,  1973).  Perception  in  chess.  Cognitive 
Psychology,  55-81. 

Chervany,  N.L. ,  &  Dickson,  G.U.  (1978).  On  the  validity  of  the  analytic- 
heuristic  instrument  utilized  in  the  "Minnesota  Experiemnts” :  A  reply. 
Management  Science.  24 .  1091-1092. 

Cohen,  M.S.,  Bromage,  R.C.,  Chinnis,  J.L.,  Payne,  J.W.,  &  Ulrila,  J.W.  (1982). 

A  personalized  and  prescriptive  attack  planning  decision  aid.  Technical 
Report  82-4.  Falls  Church,  VA:  Decision  Science  Consortium,  Inc. 

Crumley,  L.M.  (Ed.),  (1985).  Design  of  a  modular  laboratory  for  research  on 
tactical  C^.  (Abridged  version).  Working  Paper  Flv  FU  85-2.  Alexandria, 
VA:  US  Army  Research  Institute  for  the  Behaviorial  and  Social  Sciences. 

Czarnolewski ,  M.Y.  (1987).  An  information  processing  approach  for  testing 
the  validity  of  a  field  dependent  construct.  Paper  presented  at  the 
Eastern  Psychological  Association  Meetings,  9-12  April  1987. 

Arlington,  VA. 

Dede,  C,  (1986).  A  review  and  synthesis  of  recent  research  in  intelligent 
computer-assisted  instruction.  Int.  Journal  Man-Machine  Studies,  24, 
329-353. 

Doktor,  R.H.  &  Hamilton,  U.F.  (1973)  Cognitive  styles  and  the  acceptance  of 
management  science  recommendation.  Management  Science,  19 ,  884-894 

Epstein,  S.  ,  &  O'Brien,  E.J.  (1985).  The  person-situation  debate  in  historical 
and  current  perspective.  Psychological  Bulletin,  98,  513-537. 

Ericsson,  K.A.,  &  Simon,  H.A.  (1980).  Verbal  reports  as  data.  Psychological 
Review,  87 ,  215-251. 


33 


Ericsson,  K.A.,  &  Simon,  H.A.  (1984).  Protocol  analysis:  verbal  reports  on 
data.  Cambridge,  MA:  The  MIT  Press. 

Feltovich,  P.J.  (1981).  Knowledge  based  components  of  expertise  in  medical 
diagnosis .  Technical  Report  No  PDS— 2.  Learning  and  Research  Development 
Center,  University  of  Pittsburg. 

Feltovich,  P.J.  (1982).  Expertise  as  task  adaptation:  Directions  for  the 

study  of  expertise.  Learning  and  Research  Development  Center.  University 
of  Pittsburg. 

Henderson,  J.C.,  &  Nutt,  P.C.  (1980).  The  influence  of  decision  making  style 
on  decision  making  behavior.  Management  Science.  26 ,  371-386 

Henneman,  R.L. ,  &  Rouse,  W.B.  (1984).  Measures  of  human  problem  solving 

performance  in  fault  diagnosis  tasks.  IEEE  Transactions  on  Systems,  Man, 
and  Cybernetics,  SMC-14 ,  99-112. 

Huber,  G.P.  (1981).  Cognitive  style  as  a  basis  for  designing  MIS  and  DSS : 

Much  ado  about  nothing.  Working  Paper  11-81-29.  The  Graduate  School  of 
Business,  University  of  Wisconsin-Madison. 

Kuysmans,  J.H.  (1970).  The  effectiveness  of  the  cognitive  style  constraint 
in  implementing  operations  research  proposals.  Management  Science,  17, 
912-104 

Johnson,  P.E.,  Johnson,  h.G. ,  &  Little,  R.K.  (1980).  Expertise  in  trial 

advocacy:  Some  considerations  for  inquiry  into  its  nature  and  development 
Working  Paper  80-4.  Minneapolis  MN:  University  of  Minnesota,  Cognitive, 
Science  Research  Group. 

Karp,  S.A.  (1963),  Field  dependence  and  overcoming  embeddedness.  J.  Consult¬ 
ing  Psychology,  27 ,  294-302. 

Klein,  G.A.  (1981).  A  perceptual  recognitional  model  of  decision  making. 
Yellow  Springs,  OH:  Klein  Associates. 

Kolodner,  J.L.  (1984).  Towards  an  understanding  of  the  role  of  experience  in 
the  evolution  from  novice  to  expert.  In  M.J,  Coombs  (Ed.)  Developments  in 
expert  systems.  New  York:  Academic  Press,  Inc. 

Larkin,  J.H.  (1981).  Understanding,  problem  representations,  and  skill  in 

physics.  In  J.R.  Anderson  (Ed.).  Cognitive  skills  and  their  acquisition. 
Hillsdale,  N.J.:  Lawrence  Erlbaum. 

Lehner,  P.E.,  Cohen,  M.S.  Mullin,  T.M. ,  Thompson,  B.B. ,  &  Laskey,  K.B.  (1987). 
Adaptive  decision  aiding  interim  report.  Technical  Report  87-3.  Decision 


Science  Consortium,  Inc.,  Falls  Church,  VA 


Mason,  R.O.  &  Mitroff,  I.J.  (1973).  A  program  of  research  on  management 
information  systems.  Management  Science.  19 .  475-487. 

McKeown,  P.E.,  Fineberg,  M.L.,  Carter,  C.F.,  Ross,  C.G. ,  &  Packard,  B.J. 

(1985).  Command  and  control  (C^  laboratory  concept  evaluation:  final 
report «  Contract  No.  MDA903-83-C-0475.  Science  Applications  Interna¬ 
tional  Corporation,  Leavenworth,  RS. 

Moderick,  J.A.,  Levit,  R.A. ,  Alden,  D.G. ,  &  Henke,  A.H.  (1975).  Review  of 

approaches  to  cognitive  style  and  implications  for  human  information  pro¬ 
cessing  in  command  and  decision  situations.  Proceedings  of  the  human 
Factors  Society.  Santa  Monica,  CA  333-335. 

Newell,  A.  (1973).  Artificial  intelligence  and  the  concept  of  the  mind.  In. 
R.C.  Shrank  &  K.M.  Colby  (Eds).  Computer  Models  of  Language  and  Thought. 
San  Francisco,  CA:  U.H.  Freeman,  Co. 

Nisbett,  R.E.,  &  Wilson,  T.D.  (1977).  Telling  more  than  we  can  learn:  verbal 
reports  on  mental  processes.  Psychological  Review,  84 ,  231-259. 

Noah,  W.W.  ,  &  Halpin,  S.M.  (1986).  Adaptive  user  interfaces  for  planning  and 
decision  aids  in  C"^!  systems.  IEEE  Transactions  on  Systems.  Man,  and 
Cybernetics,  SMC-16,  909-918. 

Ragan,  T.J.,  Black,  K.T.,  Stonsell,  V.,  Ausburn,  L.J.,  Ausburn,  F.J. ,  Butler, 
P.A.,  Hackabay,  K.  ,  &  Burkett,  J.R.  (1978).  Cognitive  styles:  A  review  of 
the  literature.  Report  No.  AFGRLTR-78-90(l ).  HQ  Air  Force  Human  Resources 
Laboratory,  Brooks  Air  Force  Base,  Texas. 

Riedel,  S.R.  (1987),  Proceedings  of  the  Workshop  on  User  Acceptance  and  Field 
Implementation,  28-29  January  1987,  Fort  Leavenworth,  Kansas.  ARl  report. 
In  preparation. 

Robertson,  M.M. ,  &  Meshkati,  N.  (1985).  Analysis  of  the  effects  of  two 

individual  difference  classification  models  on  experiencing  mental  workload 
of  a  computer  generated  task:  a  new  perspective  to  job  design  and  task 
analysis.  Proceedings  of  the  Human  Factors  Society  -  29th  Annual  Meeting, 
pp.  178-182. 

Saga,  A.D.  (1985).  The  cognitive  model:  An  approach  to  designing  the  human 
computer  interface.  SlGCHl  Bulletin,  16,  36-40. 

Sage,  A.  (1981).  Behavioral  and  organizational  considerations  in  the  design  of 
information  systems  and  processes  for  planning  and  decision  support.  IEEE 
Transactions  on  Systems,  Han  and  Cybernetics,  SNC-1 1 ,  640-681. 

Shanteau,  J.  (1985).  Psychological  characteristics  of  expert  decision 
makers .  Psychology  Report  //85-2.  Applied  Experiemental  Psychology 
Series.  Kansas  State  University. 


35 


Sundberg,  N.D.  (1978).  The  Myers-Briggs  Type  Indicator,  In  Burcws  (Ed.)  The 
Sixth  Mental  Measurements  Yearbook. 


United  States  Army  Field  Manual,  FM  101-5  (1984).  Staff  organization  and 
operations.  Headquarters,  Department  of  the  Army. 

United  States  Army  Command  and  General  Staff  College  Reference  Book,  RB  1^1-5 
(1980).  Command  and  control  of  operations.  Fort  Leavenworth,  Kansas. 

Uitkin,  H.A.,  Ortman,  P.K. ,  Raskin,  E. ,  &  Karp,  S.A.  (1971).  A  Manual  for  the 
Embedded  Figure  Test.  Palo  Alto,  CA:  Consulting  Psychologist  Press,  Ins. 

Witkin,  H.A. ,  &  Goodenough,  D.R.  (1977).  Field  dependence  and  interpersonal 
behaviors.  Psychological  Bulletin,  84 ,  661-689. 

Znud,  R.W.  (1979).  Individual  differences  and  MIS  success:  a  review  of 
empirical  literature.  Management  Science.  25 ,  966-979. 


36 


APPENDIX  A 


Embedded  Figures  Test 


The  Embedded  Figures  Test  (EFT)  was  chosen  as  the  measure  of  cognitive 
style  in  this  experiment.  The  EFT  measures  the  global-analytic  dimension  of 
cognitive  functioning.  The  EFT  Manual  (Witkin,  et  al. ,  1977)  reviews  a  substan¬ 
tial  body  of  literature  which  suports  the  validity  of  the  use  of  the  EFT  as  an 
indicator  of  the  global-analytic  dimension  of  cognitive  functioning.  The 
global  person  tends  to  view  the  total  undifferentiated  situation,  while  the 
analytic  breaks  up  the  whole  into  the  separate  parts,  reorganizes  them  and 
recombines  them.  It  was  thus  hypothesized  that  the  global  person  would  use 
fewer  information  bits  and  prefer  less  detailed  information.  The  analytic 
person  would  use  more  information  bits  and  use  both  general  and  detailed  infor¬ 
mation.  Research  has  shown  a  relationship  between  performance  on  the  EFT  and 
in  problem  solving  tasks  where  the  solution  depends  on  using  a  critical  ele¬ 
ment  in  a  context  different  from  the  one  in  which  the  element  was  prevented, 
(Karp,  1963). 

Performance  on  the  EFT  is  not  related  to  general  intelligence  tasks,  but  is 
related  to  those  that  require  disenbedding.  Henneman  &  Rouse  (1984)  found 
that  EFT  scores  were  not  significantly  correlated  with  measures  of  cognitive 
ability,  the  American  College  Testing  Service  examines  (the  ACT),  or  the  Survey 
of  Mechanical  Insight  Esam  (SMI).  Factor  analytic  studies  have  shown  the  WAIS 
has  three  main  factors;  verbal  comprehension,  attention  concentration  and  an 
analytic  factor,  which  appears  to  have  the  task  requirement  of  separating  item 
from  context.  The  EFT  correlates  with  this  third  factor  but  not  the  first  two 
(Witkin,  et  al.,  1971).  Thus  the  EFT  is  not  related  to  full  scale  IQ  but  is 
related  to  a  component  of  the  IQ  score. 

The  EFT  measures  the  ability  to  overcome  an  embedding  context  which  then 
makes  possible  analytic  functioning;  this  is  designated  the  global-analytic  di¬ 
mension  of  cognitive  functioning. 

The  EFT  was  chosen  for  a  number  of  reasons.  First,  it  has  an  acceptable 
validity  (Witkin,  et  al.,  1977)  and  reliability.  Three  year  test-retest  relia¬ 
bility  is  .89  (Bauman,  1951).  Secondly  it  takes  a  relatively  short  time  to 
administer  -  under  30  minutes.  This  was  important  because  at  the  time  of  test¬ 
ing,  subjects  would  have  already  spent  3-4  hours  in  the  experimental  session 
and  motivation  would  be  expected  to  decline  the  longer  the  session  was  ex¬ 
tended.  Third,  the  EFT  is  content  free.  As  such  it  is  less  threatening  to  the 
subjects.  Pilot  subjects  expressed  the  concern  that  the  results  of  the  experi¬ 
ment  would  be  reported  back  to  their  superiors  or  made  part  of  their  records. 

We  are  not  sure  that  we  were  successful  in  reasuring  them  that  this  would  not 
be  the  case.  Thus,  it  was  important  to  use  a  test  that  would  not  be  suscepti¬ 
ble  to  a  social  desirability  bias,  where  subjects  respond  in  a  manner  which 
they  feel  will  look  good  to  the  Army.  The  EFT  is  not  susceptible  to  a  social 
desirability  bias. 
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Another  popular  test  of  cognitive  style  that  was  considered  for  use  in  this 
experiment  is  the  Myers  Briggs.  This  test  is  a  forced  choice  self  report  in¬ 
ventory,  designed  to  measure  four  dimensions;  judgment/perception,  thinking/ 
feeling,  extroversion-introversion,  sensation  intuitiion.  The  test  has  a 
substantial  body  of  reliability  and  validity  data.  However,  Sundberg  (1978) 
feels  that  the  purposes  of  many  of  the  questions  are  obvious  and  the  answers 
are  easily  faked.  He  says  that  a  direct  test  of  the  social  desirability  of  the 
items  needs  to  be  done  lor  the  Myers  Briggs.  Examination  of  tlie  items  suggests 
that  there  is  a  militarily  desirable  response  for  many  of  the  items,  e.g.,  "Are 
you  (a)  inclined  to  enjoy  deciding  things,  or  (b)  just  as  glad  to  have  circum¬ 
stances  decide  the  matter  for  you."  This  combined  with  the  expressed  concern 
of  several  subjects  that  the  results  of  the  procedures  would  be  reported  to 
their  superiors  or  go  into  their  records,  suggested  that  the  results  of  the 
Myers  Briggs  would  be  invalid  due  to  military  desirability  and  should  not  be 
used  as  the  test  of  cognitive  style  for  the  present  study. 

Cognitive  style  is  a  complex  concept  whose  nature  and  dimensionality  is 
unclear  despite  numerous  studies  investigating  cognitive  style.  A  great  number 
of  scales  have  been  developed  to  measure  cognitive  style,  but  the  research 
indicates  they  are  not  interchangable  and  that  they  will  vary  in  their  implica¬ 
tions  for  DSS  design  (Benbasat  &  Taylor,  1978).  For  this  reason,  an  explicit 
factor  in  choice  of  a  measure  of  cognitive  style  for  this  study  was  whether  it 
would  be  expected  to  correlate  with  information  usage  and  whether  such  a  corre¬ 
lation  would  have  implications  for  DSS  design.  Zmud  (1979)  reports  that  the 
EFT  scores  are  related  to  information  usage.  Field  independent  subjects  have 
been  found  to  seek  more  information,  prefer  detailed  aggregate  reports,  and  to 
require  more  decision  time.  Further  Benbasat  and  Taylor  (1978)  reviewing  the 
literature  on  cognitive  style,  report  that  several  studies  indicate  that  the 
field  dependent-independent  style  has  strong  implications  for  DSS  design. 

To  summarize,  all  of  the  above  factors  pointed  to  the  EFT  as  the  most 
appropriate  measure  of  cognition  style.  It  is  not  susceptible  to  social  de¬ 
sirability,  has  a  large  body  of  supporting  research,  is  widely  used,  has  ac¬ 
ceptable  validity  and  reliability,  takes  a  short  time  to  administer,  has  been 
found  to  be  related  to  information  usage  in  other  populations,  and  a  signifi¬ 
cant  relationship  between  the  EFT  and  information  usage  would  have  implications 
for  DSS  design. 
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APPENDIX  B 


Data  Base  Description* 


Data  Category  Structure 

FUNCTIONAL  AREA  DATA  CAGETORY  DATA  ELEMENT  TYPE 


PERSONNEL 

PERSONNEL  ESTlfiATE 

-  BY 

PARAGRAPH 

STRENGTHS 

-  BY 

BLUE  ORGANIZATION 

LOSSES  Ar«)  GAINS 

-  BY 

BLUE  ORGANIZATION 

OTHER  PERSONNEL 

-  BY 

BLUE  organization 

INTELLIGENCE 

INTEL  ESTIMATE 

-  BY  PARAGRAPH 

WEATHER  HISTORY 

-  5  DAYS 

WEATHER  FORECAST 

-  5  DAYS 

OPFOR  COMPOSITION 

-  BY  OPFOR  ORGANIZATION 

OPFOE  COICIITTED 

-  BY  OPFOR  ORGANIZATION 

OPFOR  REINFORCEMENTS 

-  BY  OPFOR  ORGANIZATION 

OPFOR  ARTILLERY 

-  BY  OPFOR  ORGANIZATION 

OPERATIONS 

CORPS  OP  ORDER 

-  BY  PARAGRAPH 

DIV  CMDRS  GUIDANCE 

-  CONCEPTS  &  COURSES  OF 
ACTION 

CURRENT  OPERATIONS 

-  DIVISION  &  BRIGADES 

TASK  ORGANIZATION 

-  BY  BLUE  ORGANIZATION 

LOGISTICS 

LOGISTIC  ESTIMATE 

-  BY 

PARAGRAPH 

CLASS  III  SUPPLY 

-  BY 

BLUE  ORGANIZATION 

CLASS  V  SUPPLY 

-  BY 

BLUE  ORGANIZATION 

EQUIPMENT  STATUS 

-  BY 

BLUE  ORGANIZATION 

*The  data  elements  listed  in  the  following  pages  are  those  associated  with  the 
defensive  scenario.  The  primary  difference  in  the  offensive  scenario  is  the 
specific  unit  designators,  although  there  are  a  few  differences  in  the  number 
of  data  elements. 
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Data  Elements  Within  Categories 
of  the  Personnel  Area 


PERSONNEL 


PERSONNEL 


PERSONNEL 


PERSONNEL 


LEVEL  OF 

PERSONNEL  ESTIMATE  DETAIL 


MISSION  SUMMARY 
SITUATION  SU’MMARY 
ANALYSIS  SUMMARY 
COMPARISON  SUMMARY 
CONCLUSIONS  SUMMARY 


STRENGTHS 


52  DIV  MECH 
1ST  BRIGADE 
2ND  BRIGADE 
3RD  BRIGADE 
DIVARTY 
DISCOM 
DIVTRPS 


AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 


LOSSES  AND  GAINS 


52  DIV  MECH 
1ST  BRIGADE 
2ND  BRIGADE 
3RD  BRIGADE 
52  DIVARTY 
52  DISCOM 
52  DIVTRPS 


AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 


OTHER 


52  DIV  MECH 
1ST  BRIGADE 
2ND  BRIGADE 
3RD  BRIGADE 


AGREGATED 

DETAILED 

DETAILED 

DETAILED 
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Data  Elements  within  Categories 
of  the  Intelligence  Area 


INTELLIGENCE 


INTELLIGENCE 

INTELLIGENCE 

INTELLIGENCE 


INTELLIGENCE 


INTELLIGENCE 


-  INTEL  ESTIMATE 
MISSION 

AREA  OF  OPS-WEAThER 

AREA  OF  OPS -TERRAIN 

AREA  OF  OPS-REFUGEES 

ENEMY  SIT-DISPOSITION 

ENEMY  SIT-COMPOSITION 

ENEMY  SIT-STRENGTH 

ENEMY  RECENT  ACTIVITIES 

ENEMY  PECULIARITIES/WEAKNESSES 

ENEMY  CAPABILITIES 

ANALYSIS  AND  DISCUSSION 

CONCLUSIONS 

-  WEATHER  (HISTORY) 

-  WEATHER  FORECAST 

-  OPFOR  COMPOSITION 
CENTRAL  FRONT 

10  CAA 
7  TA 
k  TA 
9  GTD 
71  GMRD 
48  GMRD 
128  MRD 

3  GTD 

6  GTD 
50  MRD 
17  TD 

-  OPFOR  COMMITED 
10  CAA 

4  TA 
9  GTD 

71  GMRD 
128  MRD 
48  GMRD 

-  OPFOR  REINFORCEMENTS 
10  CAA 

7  TA 
4  TA 
9  GTD 

71  GMRD 
128  MRD 
48  GMRD 


LEVEL  OF 

DETAIL 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUtaiARY 

SUMMARY 

AGGREGATED 

AGGREGATED 


AGGREGATED 

AGGREGATED 

AGGREGATED 

AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DE-^AILED 


AGGREGATED 

AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 


AGGREGATED 

AGGREGATED 

AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 


INTELLIGENCE  -  OPFOR  ARTILLERY 
10  CAA 
4  TA 
9  GTD 
128  MRD 
48  GMRD 
74  ARTY  REGT 
8  MRL  REGT 


AGGREGATED 

AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 
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Data  Elements  Within  Categories 
of  the  Operations  Area 


OPERATIONS  -  CORPS  OP  ORDER 
SITUATION 
MISSION 

CONCEPT  OF  OPERATION-MANEUVER 
CONCEPT  OF  OPERATION-FIRES 
MISSION-52  MECH  DIV 
MISSION-23  AR  DIV 
MISSION-201  ACR 
MISSION-FIRE  SUPPORT 
MISSION-OTHER  (SUPPORT) 
MISSION-RESERVE 
COORDINATING  INSTRUCTIONS 
SERVICE  SUPPORT 
COMMAND  AND  SIGNAL 

OPERATIONS  -  DIV  C^mRS  GUIDANCE,  5?  Tj-V  MECH 

OPERATIONS  -  CURRENT  OPERATIONS 
52  DIV  MECH 
1ST  BRIGADE 
2ND  BRIGADE 
3RD  BRIGADE 

OPERATIONS  -  TASK  ORGANIZATION 
52  DIV  MECH 
1ST  BRIGADE 
2ND  BRIGADE 
3RD  BRIGADE 
52  DIVARTY 
52  DISCOM 
52  DIVTRPS 
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LEVEL  OF 

DETAIL 

SUMMARY 

SUMMARY 

SUMMARY 

SUMMARY 

SUMl-lARY 

SUMMARY 

SU>niARY 

SUMMARY 

SUMMARY 

SUMMARY 

SLTIMARY 

SUTIMARY 

SUMMARY 

SniMARY 


AGGREGATED 

DETAILED 

DETAILED 

DETAILED 


AGGREGATED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 

DETAILED 


Data  Elements  Within  Categories 
of  the  Logistics  Area 


LOGISTICS  - 


LOGISTICS  - 


LOGISTICS  - 


LOGISTICS  - 


LOGISTICS  ESTIMATE 

LEVEL  OF 
DETAIL 

MISSION 

SUMMARY 

SITUATION 

SUTIMARY 

SITUATION-MAINTENANCE 

SUMMARY 

SITUATION-SUPPLY 

SUMMARY 

SITUATION-SERVICES 

SLTDiARY 

SITUATION-TRANSPORTATION 

SUT-IMARY 

SITUATION-ASSUMPTIONS 

SUTIMARY 

ANALYSIS-AREA  (OF  OPERATION) 

SUMMARY 

ANALYSIS -MATERIAL  &  SERVICES 

SUMMARY 

COMPARISON 

SUMMARY 

CONCLUSIONS 

SUM-IARY 

CLASS  III  SUPPLY 

52  DIV  MECH 

AGGREGATED 

1ST  BRIGADE 

DETAILED 

2ND  BRIGADE 

DETAILED 

3RD  BRIGADE 

DETAILED 

52  DIVARTY 

DETAILED 

52  DISCOM 

DETAILED 

52  DIVTRPS 

DETAILED 

CLASS  V  SUPPLY 

52  DIV  MECH 

AGGREGATED 

1ST  BRIGADE 

DETAILED 

2ND  BRIGADE 

DETAILED 

3RD  BRIGADE 

DETAILED 

52  DIVARTY 

DETAILED 

52  DISCOM 

DETAILED 

52  DIVTRPS 

DETAILED 

EQUIPMENT  STATUS 

52  DIV  MECH 

AGGREGATED 

1ST  BRIGADE 

DETAILED 

2ND  BRIGADE 

DETAILED 

3RD  BRIGADE 

DETAILED 

52  DIVARTY 

DETAILED 

52  DISCOM 

DETAILED 

52  DIVTRPS 

DETAILED 
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APPENDIX  C 


Graphics  System  Description* 


The  Graphics  Processor  provides  color  graphics  display  and  control  capa¬ 
bilities  to  the  subject  to  support  his  staff  planning  activities.  The  follow¬ 
ing  displays  are  generated  for  presentation  on  the  graphics  terminal. 

•  Digital  background  map 

•  Tactical  symbology  (including  both  unit  and  control  measure  symbology) 

Digital  Background  Map 

The  Graphics  Processor  generates  a  background  map  display  with  a  choice  of 
the  following  attributes: 

•  Vegetation 

•  Relief 

•  Sun  position 

•  Infrastructure 

•  Constant  elevation  contours 

•  UTM  grid  lines 

Each  of  these  map  attributes  are  individually  selectable  and,  except  for 
vegetation,  relief,  and  sun  position,  may  be  superimposed  (overlayed)  in  any 
combination. 

The  relief  maps  available  in  the  Defense  Mapping  Agency  (DMA)  digitized 
terrain  data  base  represents  the  terrain  applicable  to  the  selected  geographi¬ 
cal  area.  The  sun  position  button  allows  the  operator  to  activate  a  relief  map 
sun  position  feature  and  thereby  have  dynamic  control  of  sun  position  on  the 
map . 


Each  infrastructure  attribute  in  the  DMA  digitized  terrain  data  base  is 
also  represented  by  a  unique  color  as  approporate  to  adequately  reflect  terrain 
in  the  selected  geographical  area.  These  attributes  include  the  following  as 
applicable: 


2 

*  The  following  description  is  extracted  from  "Command  and  Control  (C  )  Labora¬ 
tory  Concept  Evaluation  Final  Report  (Draft):  Appendix  C,  Design  Specifica¬ 
tion,"  Science  Applications  International  Corporation  (McKeown,  1985). 


•  cities 


•  Roads/rallroads 

•  Hydrography 

•  Miscellaneous  features,  including: 

Power  stations 

-  Dans 
Tunnels 
Natural  fords 
Improved  fords 

-  Nature  surfaced  airfield/landing  zone  (AF/LZ) 
Improved  surface  AF/LZ 


Elevation  contours  are  represented  by  a  single  color  (black).  Contour 
spacing  is  tailored  to  the  map  display  level  being  displayed.  A  single  contour 
spacing  is  provided  for  each  map  display  level  as  follows: 


Map  Display  Level 
1:500,000  scale 
1:250,000  scale 
1:100,000  scale 
1:  50,000  scale 


Contour  Spacing 
None 

100  meters  in  elevation 
40  meters  in  elevation 
20  meters  in  elevation 


UTM  grid  lines  are  represented  by  single  color  (black).  UTM  grid  lines  are 
tailored  to  the  map  display  level  being  viewed  as  follows: 


Map  Display  Level 
1:500,000  scale 
1:250,000  scale 
1:100,000  scale 
1:  50,000  scale 


Grid  Spacing 
10  kilometers 
10  kilometers 
1  kilometer 
1  kilometer 


UTM  coordinate  values  are  displayed  on  each  grid  line  along  the  top  and 
left  hand  map  display  margins  at  each  display  level. 


Four  map  display  levels  are  provided  as  follows: 


Level  liap  Scale 

1  1:  50,000 

2  1:100,000 

3  1:250,000 

4  1:500,000 

The  first  three  map  display  levels  encompass  the  desired  geographical  area 
of  interest.  Map  display  levels  1-3  include  all  map  attributes  discussed 
above;  however,  only  significant  infrastructure  attributes  (e.g.,  roads, 
built-up  areas)  and  UTM  coordinates  are  displayed  for  map  level  4. 
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The  background  map  display  is  capable  of  being  zoomed  at  each  of  the  map 
display  levels.  No  additional  map  detail  is  provided  as  part  of  this  zoom 
process  (i.e.,  zoom  employs  a  pixel  explosion  technique).  The  following  three 
zoom  levels  are  provided: 

1.  x2  zoom 

2.  x4  zoom 

3.  x8  zoom 

When  in  a  zoom  configuration,  the  background  map  display  is  capable  of 
being  scrolled  left  and  right,  up  and  down,  and  at  45  degrees  angles.  The 
extent  of  the  scrolling  is  constrained  by  the  boundaries  of  the  map  display 
level  being  viewed  before  the  display  was  zoomed. 

When  at  the  bottom  of  the  map,  the  user  has  the  capability  to  either 
reposition  or  scroll  the  map  so  as  to  leave  space  for  interactive  menus. 

Generally,  background  map  display  attributes  are  each  represented  by  a 
unique  color.  A  capability  is  provided  to  select  individual  map  attribute 
colors  from  over  4,000  different  color  shades. 

Tactical  Symbology 

The  Graphics  Processor  provides  means  to  display  symbology  in  the  following 
formats : 

•  Unit  symbols  in  the  FM  21-30  format  (Figure  7) 

•  Control  measures  (Figure  8) 

BLUEFOR  (i.e.,  friendly)  unit  symbols  are  displayed  in  blue,  and  OPFOR 
(i.e.,  enemy)  unit  symbols  are  displayed  in  red.  BLUEFOR  unit  symbology  spans 
four  echelons:  company,  battalion,  brigade,  and  division.  OPFOR  unit  symbol¬ 
ogy  spans  four  echelons:  battalion,  regiment,  division,  and  combined  arms  army 
(or  comparable  level). 

Appropriate  echelon  symbols  (as  depicted  in  Figure  7),  appear  directly 
above  each  unit  symbol.  A  unit  designator  displayed  with  each  unit  symbol  and 
represents  the  unit’s  line  organization. 

A  capability  is  provided  to  define  BLUEFOR  and  OPFOR  control  measure  sym¬ 
bols  using  the  standard  symbology  specified  in  FM  21-30.  Each  of  these  control 
measures  is  color  coded  (blue  for  BLUEFOR  and  red  for  OPFOR).  Each  control 
measure  is  assigned  to  a  category  of  point,  line,  or  area.  A  line  control 
measure  may  be  comprised  of  from  two  to  eight  subject-defined  points,  and  an 
area  control  measure  may  be  comprised  of  from  three  to  eight  subject-defined 
points . 
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HE  ID  0  [o]  in 

>  t  S  &  & 


UNIT  SYMBOLS 


R  OfVtNSt 

IRMR 

RTILLIRV 

Eir-MtORCLLCD  ARTILLERY 

IRSORNE  IREAWTRY 

IR  CAVALRY 


MECHANIZED  INFANTRY  j 

medical 

anti-tank 

signal  KOMMUNICATlONSi 

infantry  j 

[^1  TRANSFORTATION 

chemical  defence 

|»0  j  ARMY  AVIATION 

armored  CAVALRY 

j/syj  electronic  WARFARE 

in  ENGINEER 

SUFFLY  AND  SERVICE 

ECHELON  SYHBOLS 


rlatoon 

•  •  • 

■RI6A0E 

comfanv 

1 

DIVISION 

EATTALION 

1  1 

regiment 

1  1  1 

MlSCELLAi^EOUS  SYMBOLS 


RIELO  MEADOUARTERS 


Figure  C-1.  UNIT  SYMBOLOGY 
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POINTS 


1.  BASIC  POINT 

•  (Label) 

2.  CHECKPOINT 

0  *(Labe1) 

3.  COORDINATION  POINT 

0  (Label) 

LINES 

1.  BASIC  LINE 

(Label ) - (Label ) 

2.  AIR  CONTROL  LINE  (ACL) 

(ACL+Label ) - (ACL+Label ) 

3.  AXIS  OF  ADVANCE 

-!?>  *(Label ) 

4.  BOUNDARY  LINE 

- II - 

UNIT/UNIT 

5.  COORDINATED  FIRE  LINE  (CFL) 

(CFL+Label ) - (CFL+Label) 

6.  DELAY  LINE  (DL) 

(DL+Label ) - (DL+Label ) 

7.  DIRECTION  OF  ATTACK 

- Label - > 

8.  FIRE  SUPPORT  COORDINATION 
LINE  (FSCL) 

( FSCL+Label ) - ( FSCL+Label ) 


4.  MAP  LEVEL 

(Label  at  designated 
coordinate) 

5.  PASSAGE  POINT 


6.  RELEASE/START  POINT 

#  (RP  +  Label  or 
SP  +  Label) 

9.  FLIGHT  ROUTE 

— > — >Label - > — > 

10.  LIMIT  OF  ADVANCE  (LOA) 

(LOA+Label ) - (LOA+Label ) 

11.  LINE  OF  CONTACT  (LC) 

(LC+Label ) - (LC+Label ) 

12.  LINE  OF  DEPARTURE  (LD) 

(LD+Label) - (LD+Label ) 

13.  MAIN  SUPPLY  ROUTE  (MSR) 

(MSR+Label )< - 


14.  PROBABLE  LINE  OF 
DEPLOYMENT  (PLD) 

(PLD+Label) - (PLD+Label) 

15.  PHASE  LINE  (PL) 

(PL+Label ) - (PL+Label) 

16.  ROUTE  OF  MARCH 

- Label - > 


Figure  C-2.  CONTROL  MEASURE  SYMBOLOGY  (Continued) 
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AREAS 


1.  BASIC  AREA  11.  MINEFIELD 


4.  AREA  OF  OPERATION  (AO) 

5.  ASSEMBLY  AREA  (AA) 

Label  ^ 

6.  DEFENSE  POSITION 

(Unit  size  on 
perimeter) 

7.  DROP  ZONE  (DZ) 


14.  OBJECTIVE  (OBJ) 


15.  OBSTACLE 
(Submenu  allows 

selection  of  four 
obstacle  types  by 
area) 

16.  PATROL  BASE 


17.  PROHIBITED  FLYING  AREA 

_l  I  I  I  I  !  L 

—  Label  — 


21. 


Figure  C-2.  CONTROL  MEASURE  SYMBOLOGY  (Concluded) 


TACTICAL  AREA  OF 
RESPONSIBILITY 
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APPENDIX  D 


Detailed  Results 

This  appendix  contains  a  detailed  description  of  the  research  findings. 
They  are  presented  in  terms  of  the  measures  used  in  the  analysis  as  listed  in 
Table  1,  page  22  of  the  text. 

Performance  Time 


Figure  D-1  shows  the  total  time  spent  in  solving  and  documenting  the  two 
tactical  problems  by  each  participant.  The  performance  times  ranged  from  1 
hour  41  minutes  to  3  hours  54  minutes  with  a  mean  of  2  hours  44  minutes  and  a 
standard  deviation  of  approximately  30  minutes.  Thus,  there  was  considerable 
variance  in  completion  times. 

There  was  no  significant  difference  in  performance  time  between  students 
and  instructors  nor  between  the  defensive  and  offensive  missions.  However,  the 
variance  in  performance  time  for  the  offensive  mission  was  significantly  great¬ 
er  than  that  for  the  defensive  mission  (F(l,14)  =  3.916,  p  <  .05). 

There  was  a  significant  difference  in  performance  times  between  the  pairs 
of  first  and  second  runs  with  the  second  run  requiring  less  time  to  complete 
(p  <  .05).  This  suggests  a  training  effect  occurred  even  with  the  training 
sessions  and  use  of  an  interactor. 

Data  Age 

Somewhat  surprisingly,  only  two  of  the  eight  participants  viewed  any  of  the 
historical  data  and  none  of  the  eight  participants  did  so  in  the  unanalyzed 
first  set  of  runs.  One  possible  explanation  for  this  is  that  the  task  did  not 
really  require  it;  the  briefing  given  before  each  run  gave  a  summary  of  what 
had  happened  to  date  and  this  may  have  been  adequate  for  most  participants. 
Other  explanations  are  that  time  and  the  amount  of  current  data  to  be  viewed 
did  not  permit  a  historical  analysis.  This  result  has  implications  for  the 
design  of  decision  support  systems  in  that  it  suggests  that,  unlike  intelli¬ 
gence  functions,  operations  personnel  may  have  relatively  little  use  for  his¬ 
torical  data. 

General  Data  Search  Patterns 


The  average  percent  of  the  total  problem  time  spent  in  data  search  was 
approximately  56%  across  all  runs  with  a  standard  deviation  of  approximately  9% 
and  range  of  42%  to  74%.  Data  search  activity  is  here  defined  as  time  spent 
viewing  the  alpha-numeric  (A/N)  data  files  and  extracting  data  from  them  as 
well  as  time  spent  viewing  the  graphics  data. 
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The  average  time  spent  in  alpha-numeric  data  search  was  approximately  36% 
and  in  graphics  search  it  was  approximately  20%.  Both,  however,  had  relatively 
large  variances  with  standard  deviations  of  about  10%  and  12%  respectively.  On 
only  three  of  the  16  runs,  did  the  amount  of  graphics  data  search  exceed  that 
of  alpha-numeric  data  search  and  one  participant  accounted  for  two  of  these  in¬ 
stances.^  There  were  no  significant  differences  between  the  defensive  and  of¬ 
fensive  mission  nor  between  students  and  instructors  nor  between  first  and 
second  runs  in  these  general  search  patterns. 


Table  D-1  indicates  how  data  search  activities  were  distributed  when  prob¬ 
lem  time  is  divided  into  quartiles.  As  would  be  expected,  the  first  quarter  of 
the  sessions  were  spent  almost  entirely  in  data  search  activities.  This  ten¬ 
dency  typically  continued  through  the  first  half  of  the  session  with  11  of  the 
16  sessions  having  90%  or  more  of  the  second  quartile  devoted  to  data  search. 

On  the  average,  approximately  78%  of  the  data  search  was  completed  by  the  mid¬ 
way  point  of  the  problem  session.  As  can  be  seen  from  Table  D-1,  there  was 
considerable  variance  among  the  sessions  concerning  the  amount  of  time  devoted 
to  data  search  during  the  last  half.  This  variance  can  be  seen  even  within 
participants  (i.e.,  between  the  first  and  second  run  for  a  given  participant). 

The  average  number  of  A/N  data  elements  viewed  (i.e.,  number  of  element 
calls)  during  a  session  was  approximately  42  and  ranged  from  17  to  68  with  a 
standard  deviation  of  13.4.  This  number  includes  many  repeat  calls  to  the  same 
element  during  a  typical  session. 


An  important  finding  concerning  the  number  of  element  calls  was  that  stu¬ 
dents  viewed  significantly  more  data  elements  than  did  the  instructors  (t  = 
4.49,  p  <  .001).  The  average  number  of  element  calls  by  students  was  53  while 
the  instructor  average  was  32.4.  Neither  the  difference  between  defense  and 
offense  nor  that  between  first  and  second  sessions  approached  significance. 

Concerning  the  level  of  detail  of  the  A/N  data  viewed,  much  more  summary 
information  was  viewed  than  was  either  aggregate  or  detailed  information.  Of 
all  the  data  element  calls  across  all  participants  and  sessions,  426  were  to 
summary  level  data,  118  to  aggregate  data  elements  and  139  were  to  detailed 
level  elements.  As  the  number  of  data  elements  available  were  not  the  same  for 
each  level  of  detail,  a  more  accurate  picture  of  level  of  detail  use  is 
obtained  by  dividing  these  figures  by  the  number  of  elements  in  each  level, 
resulting  in  the  following  averages  per  element  for  all  sessions. 


The  participant  who  spent  the  most  time  in  graphics  data  search  did  a  thor¬ 
ough  job  of  wargaming  the  possible  courses  of  action  using  the  graphics  dis¬ 
play.  It  should,  however,  be  noted  that  time  was  counted  as  involving  graphics 
search  only  from  the  time  some  action  was  taken  on  the  graphics  display  in 
other  than  the  overlay  creation  mode  until  some  other  type  of  action  was  taken. 
Although  this  appeared  to  be  a  generally  satisfactory  criterion,  some  periods 
of  graphics  search  were  missed  in  which  no  new  graphics  actions  were  taken. 


D-3 


Table  D-1 


Percent  of  Experiment  Session  Quartile  Time  Spent  in  Data  Search  Activities 

Students* 


Quartile 

ASH 

AS12 

AS21 

AS22 

BSll 

BS12 

BS21 

BS22 

1 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

90.2 

100.0 

2 

100.0 

97.8 

48.8 

56.5 

100.0 

100.0 

100.0 

100.0 

3 

5.9 

39.1 

28.0 

43.2 

27.1 

17.9 

63.6 

66.9 

4 

11.5 

0.0 

13.1 

19.3 

0.0 

9.2 

11.6 

18.5 

Instructors* 

Quartile 

Alll 

AI12 

AI21 

AI22 

Bill 

BI12 

BI21 

BI22 

1 

96.6 

100.0 

100.0 

92.9 

100.0 

100.0 

100.0 

100.0 

2 

90.9 

73.7 

70.3 

81.0 

93.4 

93.6 

100.0 

100.0 

3 

48.8 

6.5 

3.6 

7.9 

43.6 

28.3 

100.0 

30.8 

4 

25.9 

0.0 

46.5 

44.4 

0.0 

74.5 

20.0 

11.0 

Overall 

Quartile _ Means _ Standard  Deviation 


1 

98.73 

2.92 

2 

87.87 

16.36 

3 

35.07 

25.32 

4 

19.09 

19.82 

*Participants  codes  are  interpreted  as  follows: 

First  letter  is  the  sequence  code.  A  =  Defense  first  run,  offense  second  run. 

B  =  Offense  first  run,  defense  second  run. 
Second  letter  is  the  participant  type  code. 

S  =  Student 
1  =  Instructor 

First  number  identifies  the  participant  within  the  sequence/subject  type 
category  (1  or  2).  Second  number  identifies  which  run  it  is  for  that 
participant  (1  or  2). 
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Summary  10.3 

Aggregate  5.6 

Detailed  2.2 

A  clear  progression  is  thus  evident  from  detailed  to  summary  level  informa¬ 
tion.  This  suggests  that  participants  tended  to  rely  on  the  conclusions  of 
others,  and  certainly  did  not  seek  confirming  information  in  the  majority  of 
instances.  When  numeric  information  was  required,  they  tended  to  view  these 
data  as  aggregated  for  the  division  or  at  brigade  level,  occasionally  looking 
at  individual  battalions  if  warranted  (e.g.,  a  unit  had  suffered  heavy  losses 
or  the  proposed  division  task  organization  might  require  its  reorganization). 

One  of  the  few  significant  differences  among  experimental  groups  was  found 
in  the  area  of  level  of  detail  usage,  lifhen  viewed  as  a  percentage  of  the  total 
number  of  element  calls  per  participant,  instructors  looked  at  more  summary 
level  data  (t  =  3.U5,  p  <  .01)  and  less  detailed  data  (t  =  2.42,  p  <  .05)  than 
did  the  students.  Figure  D-2  shows  the  distribution  of  element  calls  for  the 
two  groups. 

Functional  Area  Use 


Alpha-numeric  (A/N)  information  available  in  the  data  base  was  categorized 
into  the  four  functional  areas  of  personnel,  intelligence,  operations,  and  lo¬ 
gistics.  Figure  D-3a  shows  the  average  percent  of  A/N  data  search  time  that 
was  spent  in  each  functional  area  across  all  sessions.  Since  the  number  of 
data  elements  differed  among  the  four  functional  areas,  the  percent  of  time 
divided  by  the  number  of  data  elements  is  probably  a  better  indication  of  the 
relative  use  of  the  functional  areas.  Figure  D-3b  reflects  these  values. 

On  the  average,  about  77%  of  the  A/N  data  search  time  was  spent  looking  at 
operations  and  intelligence  data  elements.  However,  there  was  considerable 
difference  among  the  16  sessions  as  to  which  of  these  two  functional  areas  were 
viewed  the  most  and  the  differences  were  not  related  to  students  vs.  instruc¬ 
tors,  defense  vs.  offense,  or  first  session  vs.  second  session.  Indeed,  there 
were  no  significant  differences  in  usage  in  any  of  these  three  variables  for 
any  of  the  four  functional  areas. 

Probably  most  surprising  is  the  lack  of  consistency  between  the  two  ses¬ 
sions  for  given  individuals.  Tnere  was  no  significant  relationship  between  the 
rank  order  of  individuals  on  the  defensive  and  offensive  missions  using 
Kendall's  Tau  concerning  use  of  any  of  the  four  functional  areas.  When  we 
considered  how  good  a  predictor  the  percentage  score  on  one  session  was  of  the 
percentage  score  of  the  individual  on  the  second  session  compared  with  that  of 
all  other  subjects  on  the  second  session,  we  found  that,  overall,  the  predic¬ 
tions  were  no  better  than  might  be  expected  if  the  values  had  been  assigned  to 
the  participants  randomly. 
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There  was  however,  a  definite  pattern  for  the  sequence  in  which  the  func¬ 
tional  areas  were  viewed  as  shown  in  Table  D-2  below.  The  numbers  in  the  body 
of  the  table  are  the  number  of  sessions  in  which  that  functional  area  was 
viewed  in  the  order  given.  For  example,  in  nine  sessions  data  from  the  per¬ 
sonnel  area  was  the  third  functional  area  viewed.  Thus  the  table  reflects  the 
existence  of  a  typical,  or  modal  pattern  of  sequential  functional  area  viewing. 


Table  D-2 


Sequential  Viewing  of  Functional  Areas 


Functional 

Area 

1st 

2nd 

3rd 

4  th 

5  th 

6th 

7th 

8th 

9th 

10  th 

11th  12th  13th 

Personnel 

3 

4 

(D 

4 

1 

2 

1 

1 

1 

Intelligence 

© 

2 

4 

© 

2 

2 

1 

2 

Operations 

O 

2 

2 

2 

© 

3 

5 

1 

2 

1  1 

Logistics 

3 

3 

© 

3 

1 

1 

The  circles  indicate 

that 

the 

typical  participant 

viewed  operations  data 

first  then  intelligence. 

personnel 

,  and 

logistics 

data 

in 

order 

followed  by  a 

return  to  either  intelligence  or  operations.  To  interpret  in  operational 
terms,  the  typical  participant  first  looked  at  the  mission  requirements  and 
commander's  guidance  plus  the  status  of  his  own  forces.  He  then  studied  the 
terrain  and  the  enemy  forces.  He  then  went  back  to  look  at  supporting  data  he 
felt  were  important,  taking  them  in  order  beginning  with  personnel.  Following 
this,  he  would  return  to  the  operations  or  intelligence  data  to  confirm  infor¬ 
mation  important  to  the  concept  he  was  developing. 

Figure  D-4  depicts  the  use  of  functional  area  data  in  another  way.  It 
shows  the  average  (N=16)  accumulation  of  functional  area  information  over  a 
session  period.  Almost  half  of  the  textual  operations  information  was  viewed 
in  the  first  tenth  of  the  average  session,  again  reflecting  the  gathering  of 
mission  information  early  in  the  session.  It  should  be  noted  that  fairly  large 
individual  differences  existed  in  how  functional  area  viewing  was  distributed 
over  time  but  these  differences  were  not  attributable  to  the  student/instruc¬ 
tor,  defense/offense,  or  first  run/second  run  variables. 

The  same  was  true  for  the  distribution  of  note  taking  from  the  four  func¬ 
tional  areas;  large  individual  differences  but  no  significant  group  differ¬ 
ences.  Across  all  participants  and  sessions  about  40%  each  of  the  notes  were 
taken  from  operations  and  intelligence  data  and  about  10%  each  were  taken  from 
personnel  and  logistics  data. 
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Concerning  the  relationship  of  level  of  detail  to  functional  areas,  in  all 
but  the  personnel  area  the  majority  of  element  calls  were  to  summary  level  data 
elements.  Table  D-3  shows  the  total  number  of  element  calls  across  all  ses¬ 
sions  for  each  level  of  detail  within  each  functional  area. 


Table  D-3 
Level  of  Detail 


Functional  Area 

Summary 

Aggregate 

Detailed 

Personnel 

39  (  7.8) 

11  (  3.7) 

42  (2.8) 

Intelligence 

125  (10.9) 

46  (  3.5) 

45  (2.1) 

Operations 

199  (14.2) 

38  (19.0) 

23  (2.6) 

Logistics 

63  (  5.7) 

24  (  8.0) 

28  (1.6) 

The  figures  in  parentheses  represent  the  results,  as  a  percentage,  of  di¬ 
viding  each  total  by  the  number  of  data  elements  available  for  viewing  in  that 
category.  This  yields  a  more  accurate  picture  of  the  relative  distribution  of 
element  calls.  It  will  be  noted  that  these  parenthetical  values  increase  the 
relative  frequency  of  aggregate  data  use  while  decreasing  that  of  detailed 
data.  The  large  parenthetical  value  for  aggregate  operations  data  results  from 
the  frequent  viewing  of  the  division-level  current  task  organization  data  ele¬ 
ment.  The  very  low  parenthetical  value  for  detailed  logistics  data  probably 
reflects  the  absence  of  any  substantial  logistical  problems  in  either  scenario. 

Data  Category  Use 

Each  of  the  four  functional  areas  of  A/N  data  contained  different  catego¬ 
ries  of  information.  These  categories  represent  distinct  types  of  information 
and  were  typically  analogous  to  various  types  of  reports  such  as  staff  esti¬ 
mates  and  status  reports.  Table  D-4  shows  these  categories  and  characteristics 
of  their  use  in  the  experiment.  An  explanation  of  the  content  of  these  data 
categories  is  contained  in  Appendix  B. 

As  can  be  seen  from  the  table,  only  three  data  categories  were  looked  at  in 
every  session  and  11  of  the  19  categories  were  viewed  in  over  half  of  the  ses¬ 
sions.  The  least  used  categories  were  those  consisting  of  information  that  was 
either  relatively  narrow  in  application  and/or  historical  in  nature  (e.g.. 

Other  Personnel  or  Weather  History)  or  whose  content  could  be  obtained  in  sum¬ 
mary  form  in  the  staff  estimates  (e.g..  Enemy  Composition  or  Class  V  Supplies). 

Not  surprisingly,  the  data  categories  viewed  the  most  frequently  for  the 
longest  duration  and  from  which  the  most  data  were  extracted  were  the  staff 
estimates,  the  mission  data  (i.e..  Corps  Operations  Order  and  Division  Com¬ 
mander's  Guidance)  and  the  Task  Organization.  These  are  the  data  categories 
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Table  D-4 


Data  Category  Use 

Percent  of 

Average 

Average  if 

Functional 

Sessions  in 

Time  Viewed 

of  Notes 

Area  Data  Category 

Which  Used 

per  Session 

per  Session 

Personnel 

Personnel  (Gl)  Estimate 

87.5 

2:32 

1.4 

Strengths 

75.0 

2:30 

1.0 

Losses  and  Gains 

18.7 

:14 

0 

Other  Personnel 

12.5 

:04 

0 

Intelligence 

Intelligence  (G2)  Estimate 

100.0 

17:37 

9.1 

Weather  History 

6.2 

:03 

0 

Weather  Forecast 

50.0 

:42 

.4 

Enemy  Composition 

43.8 

:47 

.2 

Enemy  Committed 

75.0 

1:52 

.5 

Enemy  Reinforcements 

75.0 

1:47 

.6 

Enemy  Artillery 

31.2 

:34 

0 

Operations 

Corps  Operations  Order 

100.0 

9:12 

5.2 

Div  Commander's  Guidance 

93.7 

6:30 

4.3 

Current  Operations 

75.0 

1:58 

0 

Task  Organizations 

93.7 

3:22 

.7 

Logistics 

Logistics  (G4)  Estimate 

100.0 

3:27 

1.6 

Class  III  Supply 

50.0 

:17 

0 

Class  V  Supply 

43.8 

:11 

0 

Equipment  Status 

75.0 

3:14 

1.0 
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most  directly  related  to  the  task  of  developing  an  operations  estimate  and, 
when  time  is  limited,  provide  summary  data  that  covers  most  aspects  of  the 
situation. 

It  should  be  noted  that  the  Intelligence  Estimate  was  viewed  for  longer 
periods  and  more  notes  were  taken  from  it  than  any  other  data  category.  The 
intelligence  estimate,  however,  consists  of  either  16  or  19  (i.e.,  scenario 
dependent)  pages  of  purely  textual  information.  In  comparison,  the  Task  Or¬ 
ganization  which  was  actually  viewed  more  frequently  than  the  Intelligence 
Estimate  (average  2.9  times  per  session  vs.  2.7)  contained  only  seven  pages  of 
purely  tabular  data. 

It  can  be  seen  from  Table  D-5  that  the  instructors  concentrated  their  data 
search  on  the  summary  information  in  the  staff  estimates.  Corps  Operations 
Order  and  Division  Commander's  Guidance.  The  only  other  categories  viewed  in 
more  than  half  of  the  instructor  sessions  were  Task  Organization  and  (person¬ 
nel)  Strengths.  Note  the  extreme  difference  between  students  and  instructors 
in  viewing  the  Class  III  and  V  supply  data. 

There  were  no  differences  between  defensive  and  offensive  scenarios  nor  be¬ 
tween  first  and  second  sessions  in  the  use  of  data  categories. 

Data  Element  Use 


There  were  114  data  elements  common  to  both  scenarios  plus  an  additional  13 
data  elements  that  were  scenario-unique.  A  data  element  is  a  subclassification 
of  a  data  category  that  contains  information  relating  to  a  particular  aspect  of 
the  data  category.  For  example,  the  logistical  data  category  of  "equipment 
status"  is  divided  into  seven  data  elements,  one  containing  equipment  status 
information  for  the  division  as  a  whole  and  the  other  six  containing  more  de¬ 
tailed  information  for  separate  organizations  within  the  division.  A  listing 
of  the  data  elements  is  contained  in  Appendix  B. 

There  was  considerable  variation  among  the  16  experimental  sessions  as  to 

the  data  elements  viewed.  Of  the  114  common  data  elements,  36  were  viewed  in 

one-half  or  more  of  the  sessions  and  15  of  these  were  viewed  in  IbX  or  more  of 
the  sessions.  There  were  only  two  data  elements  that  were  viewed  in  all  16 
sessions,  the  corps  mission  statement  in  the  Corps  OPORD  and  the  Division  Com¬ 
mander's  Guidance;  this  despite  the  fact  that  the  average  number  of  unique 
data  elements  viewed  per  session  was  37.7  (SD  =  11.68). 

Table  D-6  shows  the  15  data  elements  that  were  viewed  most  often.  Over 

half  of  these  are  from  the  Corps  Operations  Order  (OPORD)  data  category.  Of 

the  five  data  elements  in  the  list  that  were  not  from  the  operations  functional 
area,  three  of  them  are  the  conclusion  sections  of  the  other  staff  estimates. 


2 

of 


The  Division  Commander's  Guidance  was  actually  a  data  category  consisting 
four  pages  of  text  that  was  not  further  broken  down  into  data  elements. 
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Table  D-5 


Conparison  of  Data  Category  Use  Between  Students  and  Instructors 


Data  Category 

Percent  of  Sessions 

Students 

in  Which  Viewed 

Instructors 

Personnel  (Gl)  Estimate 

75 

100 

Strengths 

75 

75 

Losses  and  Gains 

37.5 

0 

Other  Personnel 

25 

0 

Intelligence  (G2)  Estimate 

100 

100 

Weather  History 

12.5 

0 

Weather  Forecast 

75 

25 

Enemy  Composition 

75 

12.5 

Enemy  Committed 

100 

50 

Enemy  Reinforcements 

100 

50 

Enemy  Artillery 

50 

0 

Corps  Operations  Order 

100 

100 

Div  Commander’s  Guidance 

100 

87.5 

Current  Operations 

100 

50 

Task  Organizations 

100 

87.5 

Logistics  (G4)  Estimate 

100 

100 

Class  III  Supply 

100 

0 

Class  V  Supply 

100 

0 

Equipment  Status 

100 

50 

D-13 


I 

I 

I 

■ 


Table  D-6 

Most  Commonly  Viewed  Data  Elements 


Percent  of  Sessions 

Data  Element _  _  in  which  Viewed 


Division  Commaner's  Guidance  100.0 
Corps  OPORD  -  Corps  Mission  100.0 
Corps  OPORD  -  Concept  of  Operation,  Maneuver  93.8 
Corps  OPORD  -  Concept  of  Operation,  Fires  93.8 
Task  Organization  -  Division  93.8 
Intelligence  Estimate  -  Area  of  Operations,  Weather  87.5 
Corps  OPORD  -  Own  Division's  Mission  87.5 
Corps  OPORD  -  Fire  Support  Mission  87.5 
Corps  OPORD  -  Reserve  Mission  87.5 
Logistics  Estimate  -  Conclusions  87.5 
Personnel  Estimate  -  Conclusions  81.3 
Intelligence  Estimate  -  Area  of  Operations,  Terrain  81.3 
Corps  OPORD  -  Coordinating  Instructions  81.3 
Intelligence  Estimate  -  Conclusions  75.0 
Corps  OPORD  -  Situation  75.0 
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There  were  27  data  elements  that  were  not  viewed  in  any  of  the  16  sessions. 
These  are  listed  in  Table  D-7.  Nine  of  these  were  in  the  personnel  functional 
area,  five  in  intelligence,  one  in  operations  and  12  in  logistics  (see  Appendix 
F  for  a  list  of  these).  Eight  of  the  nine  not  viewed  personnel  data  elements 
were  in  the  data  categories  of  Personnel  Losses  &  Gains  and  the  Personnel  Other 
category  containing  prisoner  of  war  (POU)  capture  and  holding  status  informa¬ 
tion.  The  five  intelligence  data  elements  all  deal  with  the  organization  of 
specific  enemy  units,  three  being  Front  level  second  echelon  units  and  two 
being  Army  level  artillery  regiments.  The  one  data  element  not  viewed  in  the 
operations  area  was  the  task  organization  of  the  division  artillery  (DIVARTY). 
Because  of  a  system  limitation  the  DIVARTY  task  organization  did  not  reflect 
the  cross  attachment  of  its  battalions,  thus  making  it  of  limited  value.  All 
but  one  of  the  twelve  logistics  data  elements  not  viewed  were  Class  III  (ammu¬ 
nition)  and  Class  V  (fuel)  status  reports  on  subelements  of  the  division. 

There  were  no  supply  problems  in  either  scenario;  a  situation  which  could  be 
readily  ascertained  by  viewing  the  division  level  Class  III  &  V  status.  It 
should  be  noted  that  all  but  one  of  the  elements  not  viewed  were  at  the 
detailed  level  of  detail. 

The  effects  of  the  scenario  on  which  of  the  data  elements  are  viewed  appear 
to  be  limited  in  this  experiment.  There  were  only  seven  data  elements  which 
differed  across  scenarios  as  to  the  number  of  participants  viewing  by  four  or 
more.  Four  of  these,  in  which  more  frequent  viewing  occurs  in  the  defensive 
scenario,  are  elements  contained  in  the  last  half  of  the  intelligence  estimate. 
As  there  is  no  apparent  reason  for  their  more  frequent  use  in  the  defense,  it 
is  likely  that  this  is  a  result  of  differences  in  the  general  amount  of  infor¬ 
mation  contained  in  the  two  estimates.  The  other  elements  are  more  apt  to  be 
related  to  mission  type.  More  participants  looked  at  the  "intelligence  esti¬ 
mate,  area  of  operations  -  refugees"  data  element  in  the  defense  then  in  the 
offense,  perhaps  reflecting  a  greater  perceived  effect  on  delaying  operations. 
More  participants  looked  at  the  weather  forecast  in  the  offensive  scenario 
perhaps  because  of  its  potential  effect  on  timing  of  offensive  operations. 

Also,  more  participants  looked  at  the  enemy  artillery  status  at  division  level 
in  the  offensive  scenario  perhaps  because  of  its  greater  potential  to  hinder 
offensive  operations. 

One  of  the  most  interesting  aspects  of  data  element  usage  is  the  within 
subject  use  of  data  elements  across  the  two  sessions.  As  the  type  of  scenario 
seems  to  have  little  affect,  we  might  expect  consistency  within  individuals  in 
what  data  elements  they  viewed  in  the  two  sessions.  To  investigate  this,  we 
compared  the  ratio  of  same  elements  ("matches")  to  dissimilar  elements  ("mis¬ 
matches")  viewed  in  the  two  sessions  for  each  of  the  eight  participants.  Four 
of  the  participants  had  more  matches  than  mismatches  and  over  all  the  partici¬ 
pants  the  ratios  ranged  from  one  match  for  every  .4A  mismatches  to  one  match 
for  every  4.13  mismatches.  The  students  and  instructors  were  evenly  distrib¬ 
uted  over  this  range. 

To  get  some  idea  as  to  what  this  means  in  more  absolute  terms  we  compared 
the  within-subject  ratios  to  the  between-subject  ratios  (i.e.,  participant's 
defense-offense  matching  ratio  compared  to  the  ratio  of  his  defense  element  se¬ 
lection  to  all  other  participant's  offense  element  selection  and  visa-versa). 
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Table  D-7 


Data  Elements  Not  Viewed  in  Any  Session _ 

Functional  Data  Data 

Area _ Category  Element _ Notes _ 

Personnel  -  Strengths  -  DISCOM 
Personnel  -  Losses  &  Gains  -  2  Bde 

Personnel  -  Losses  &  Gains  -  3  Bde 

Personnel  -  Losses  &  Gains  -  DIVARTY 
Personnel  -  Losses  &  Gains  -  DISCOM 
Personnel  -  Losses  &  Gains  -  DIVTRPS 

Personnel  -  Other  -  1  Bde 

Personnel  -  Other  -  2  Bde 

Personnel  -  Other  -  3  Bde 

Intelligence  -  OPFOR  Composicion-*3GTD/l IIMRD  (Front  second  echelon  divisions) 

Intelligence  -  OPFOR  Composition-*! 7TD/4TD  (Front  second  echelon  divisions) 

Intelligence  -  OPFOR  Reinforcements  -  /14TA  (Front  second  echelon  divisions) 

Intelligence  -  OPFOR  Artillery-/79  Arty  Regt  (Front  second  echelon  divisions) 

Intelligence  -  OPFOR  Artillery-/58  Arty  Regt  (Front  second  echelon  divisions) 

Operations  -  Task  Organization  -  DIVARTY 

Logistics  -  Class  III  Supplies  -  2  Bde 

Logistics  -  Class  III  Supplies  -  3  Bde 

Logistics  -  Class  III  Supplies  -  DIVARTY 

Logistics  -  Class  111  Supplies  -  DISCOM 

Logistics  -  Class  III  Supplies  -  DIVTRPS 

Logistics  -  Class  V  Supplies  -  1  Bde 

Logistics  -  Class  V  Supplies  -  2  Bde 

Logistics  -  Class  V  Supplies  -  3  Bde 

Logistics  -  Class  V  Supplies  -  DIVARTY 

Logistics  -  Class  V  Supplies  -  DISCOM 

Logistics  -  Class  V  Supplies  -  DIVTRPS 

Logistics  -  Equipment  Status  -  DISCOM 


*  First  unit  is  from  the  defensive  scenario,  second  is  from  the  offensive  sce¬ 
nario.  They  are  matched  upon  their  positions  in  the  enemy  order  of  battle. 
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Although  the  within-sub ject  ratio  was  better  (i.e.,  lower)  than  the  between- 
subject  ratios  in  81  of  the  112  comparisons,  none  of  the  16  z-values  were  sig¬ 
nificant  nor  were  the  means  of  the  within-subject  and  between-subject  ratios 
significantly  different.  This  suggests  that  participants  have  no  set  schema 
for  attacking  the  problem  that  clearly  differentiates  them  from  others  solving 
the  same  problem. 

Graphics  Data  Use 

Graphics  data  could  be  viewed  on  any  of  four  scales  corresponding  to  the 
amount  of  terrain  contained  in  the  graphics  display  and,  inversely,  the  amount 
of  terrain  detail  available.  To  summarize: 

Scale  Approximate  Area  Displayed  (in  Kilometers) 


1:50,000 

1:100,000 

1:250,000 

1:500,000 


10  X  10 
20  X  20 
50  X  50 
100  X  100 


The  1:50,000  and  1:100,000  scales  were  used  the  exist  accounting  for  approx¬ 
imately  31%  and  46%  of  the  graphics  viewing  time  respectively  over  all 
sessions.  The  1:250,000  scale  was  used  22%  of  the  time  while  the  1:500,000 
scale  was  used  less  than  1%  of  the  time.  The  predominance  of  the  1:100,000  is 
probably  because  it  displayed  the  entire  division  frontline  trace,  or  nearly 
so,  while  still  allowing  sufficient  terrain  detail  for  division  level  planning. 
The  1:50,000  scale,  which  displays  approximately  a  brigade  frontline  trace  was 
used  primarily  when  analyzing  the  terrain.  The  large  scales  were  difficult  to 
use  because  of  the  clutter  when  unit  symbols  were  displayed  and  the  lack  of 
detail  concerning  terrain  features.  The  zoom  capability,  which  allowed  the 
participants  to  magnify  an  area  of  the  display  by  2,  4,  or  8  times  was  rarely 
used;  one  possible  reason  being  that  it  added  no  additional  detail  to  the  scale 
being  used.  There  were  no  significant  differences  between  students  and 
instructors  nor  defensive  and  offensive  scenarios  in  the  use  of  graphics 
scales . 


A  common  problem  voiced  by  all  the  participants  was  in  maintaining  their 
orientation  when  using  the  graphics  display.  Unlike  the  large  wall  maps  they 
were  used  to,  they  could  not  step  back  and  see  the  "big  picture."  For  example, 
when  moving  around  the  terrain  at  a  1:50,000  resolution  it  was  difficult  to 
remember  exactly  where  they  were  in  relation  to,  say,  the  frontline  or  some 
other  object.  Also,  they  found  it  difficult  to  remember  the  objects  and  their 
locations  that  were  adjacent  to  the  displayed  area. 

Two  main  types  of  terrain  background  were  available.  The  vegetation  back¬ 
ground  displayed  the  type  of  vegetation  characterizing  the  terrain  (e.g.,  open 
farmland,  water,  marsh,  evergreen  forest,  etc.),  using  color  codes.  The  relief 
background  used  shades  of  brown  to  represent  elevation  and  dark  shading,  repre¬ 
senting  shadows,  to  indicate  the  pitch  of  slopes.  These  were  mutually  exclu¬ 
sive  backgrounds  and  the  participant  could  also  choose  to  have  no  background  at 
all. 
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The  relief  background  was  used  very  little,  only  about  2%  of  the  time  over 
all  sessions;  perhaps  because  it  differed  considerably  from  the  types  of  wall 
maps  the  participants  were  used  to  and  also  because  relief  could  be  indicated 
by  displaying  contour  lines  regardless  of  the  type  of  terrain  background  used. 
Because  four  of  the  sessions  used  vegetation  almost  exclusively  the  overall  av¬ 
erage  showed  a  preponderant  use  of  vegetation  as  compared  to  no  terrain  back¬ 
ground  (2:1).  However,  looking  at  the  individual  sessions,  half  of  them  used 
no  terrain  background  more  frequently  than  they  used  the  vegetation  background. 
For  some  of  the  participants,  clutter  was  a  real  annoyance,  and  removing  the 
terrain  background  when  it  was  not  necessary  reduced  the  clutter  considerably. 

The  special  display  features  available  include  hydrology,  cities,  roads/ 
railroads,  contours,  grids,  and  miscellaneous  (i.e.,  bridges,  fords,  tunnels, 
and  airfields).  Of  the  six,  the  UTM  location  grids  were  used  the  most  often 
followed  by  the  roads/railroads,  contours,  hydrology  and  cities  in  that  order. 
The  miscellaneous  features  were  used  in  only  four  sessions,  all  of  them  in  the 
offensive  scenario  where  bridge  crossing  sites  were  important. 

There  was  wide  variance  among  the  sessions  concerning  the  viewing  of  unit 
tactical  symbology.  On  the  average,  participants  had  unit  symbols  displayed 
about  half  the  time  they  were  viewing  the  graphics,  but  the  range  across  ses¬ 
sions  was  considerable  (i.e.,  6%  to  95%).  Typically,  however,  the  participants 
kept  the  unit  boundary  lines  on  all  the  time  they  were  viewing  graphics.  This 
allowed  them  to  view  the  tactical  situation  (i.e.,  the  boundary  line  displays 
contained  the  unit  designators)  without  the  clutter  caused  by  the  unit  symbols. 

Working  File  Use 

As  was  stated  previously,  the  working  file  or  electronic  note  pad  was  or¬ 
ganized  by  the  participant.  He  outlined  it  to  contain  the  topics  he  wanted  to 
cover.  He  then  entered  notes  he  wished  to  take  under  each  topic  either  by 
transferring  lines  directly  from  the  reports  terminal  displays  or  by  summariz¬ 
ing  information  in  his  own  words. 

There  were  considerable  differences  among  the  participants  as  to  the  spe¬ 
cific  titles  they  used  in  outlining  their  working  files  but  the  majority  con¬ 
tained  some  reference  to  all  of  the  METT  factors  (i.e.,  mission,  enemy, 
terrain,  and  troops).  In  four  of  the  sessions,  however,  very  generic  catego¬ 
ries  were  used;  one  participant,  for  example,  used  only  two  categories  in  his 
sessions,  "notes"  and  "questions"  or  "key  points."  Participants  tended  to  use 
fewer  categories  to  outline  their  working  files  in  the  second  session,  but  the 
difference  was  not  significant  (Xj^  =  13.0,  X2  =  9.9,  t  =  2.18,  p  >  .05).  There 
were  no  differences  between  the  students  and  instructors  nor  the  defensive  and 
offensive  scenarios  in  the  number  of  categories  used  in  outlining. 


There  also  was  considerable  variance  between  the  sessions  concerning  the 
specific  notes  that  were  taken.  In  the  personnel  and  logistics  functional 
areas,  only  the  "conclusions"  portion  of  the  estimates  was  noted  in  at  least 
half  the  sessions.  In  all  of  these  (i.e,,  eight  sessions  for  the  personnel 
estimates  conclusions,  and  ten  for  the  logistics  estimate  conclusions)  the 


participants  extracted  at  least  that  portion  which  stated  the  ordering  of  po¬ 
tential  courses  of  action  as  to  their  supportability  from  the  personnel  and 
logistics  standpoints. 

In  the  intelligence  functional  area,  the  intelligence  estimate  was  heavily 
noted,  but  only  the  data  elements  of  "area  of  operations-weather"  and  "area  of 
operations-terrain"  were  noted  in  at  least  half  the  sessions.  Notes  taken  from 
"area  of  operations-weather"  in  the  defense  primarily  involved  the  fact  that 
rain  expected  in  three  days  should  slow  the  enemy;  in  the  offense,  that  wind 
conditions  favor  the  use  of  smoke  by  friendly  forces.  In  the  "area  of 
operations-terrain"  data  element  in  both  the  defense  and  offense  many  notes 
were  taken  concerning  obstacles,  key  terrain,  and  friendly  effects  of  terrain. 
In  addition,  in  the  offense,  many  notes  were  taken  concerning  avenues  of 
approach  and  cover  and  concealment  which  was  not  true  in  the  defense.  Within 
these  general  terrain  topic  areas,  however,  considerable  variance  existed  as  to 
the  specific  features  noted. 

In  the  operations  functional  area,  both  the  corps  operations  order  and  the 
division  commanders  guidance  data  categories  were  heavily  noted.  In  the  corps 
operations  order,  the  "corps  mission"  and  "own  division  mission"  data  elements 
were  noted  in  at  least  half  the  sessions,  typically  by  extracting  the  entire 
statement.  The  "concept  of  operation-maneuver"  data  element  was  noted  in  9  of 
the  16  sessions;  in  both  defense  and  offense  what  was  typically  extracted  was 
the  concept  statement  for  the  participant's  own  division.  The  "mission-fire 
support"  data  element  was  noted  in  12  sessions;  although  the  air  support  pri¬ 
orities  and  air  assets  allocations  were  frequently  noted  the  most  commonly 
noted  aspect  of  "mission-fire  support"  was  the  field  artillery  (FA)  organiza¬ 
tion  for  combat  which  showed  the  attachment  of  corps  FA  units  to  its  divisions. 
An  interesting  phenomenon  is  the  more  frequent  note  taking  from  the  "concept  of 
operations-f ires"  and  "mission-reserve"  data  elements  in  the  offense  than  in 
the  defense.  In  the  offense,  the  priority  of  commitment  of  these  assets  was 
to  the  participant's  division;  in  the  defense,  his  division  had  low  priority 
for  receiving  these  assets. 

The  division  commander's  guidance  data  category  was  not  further  broken  down 
into  data  elements ,  but  note  taking  can  be  discussed  in  terms  of  the  general 
topics  that  were  noted.  The  most  widely  noted  of  any  topic  throughout  the 
entire  data  base  was  the  description  of  the  possible  courses  of  action  con¬ 
tained  in  the  division  commander's  guidance.  In  14  of  the  16  sessions,  infor¬ 
mation  was  noted  on  this  topic,  typically  a  verbatim  extract  of  the  courses  of 
action.  The  other  commander's  guidance  topic  noted  in  at  least  half  the  ses¬ 
sions  was  the  enemy's  potential  for  using  nuclear  and  chemical  weapons.  In  10 
sessions  Che  subjects  took  notes  concerning  the  division  commander's  reminder 
to  take  this  into  account  in  their  planning.  Other  topics  frequently  noted 
were  scenario-specific.  For  instance,  the  discussion  of  the  critical  enemy 
avenues  of  approach  and  the  need  for  rear  area  security  that  appeared  in  the 
defensive  scenario  commander's  guidance  were  noted  by  the  majority  of  parti¬ 
cipants.  In  the  offense,  it  was  the  discussion  of  the  general  attack  concept. 
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As  was  done  with  data  element  viewing,  we  attempted  to  measure  the  consis¬ 
tency  of  individual  participants  in  what  topics  they  took  notes  from  between 
their  first  and  second  sessions.  Again  this  was  done  by  analyzing  the  ratio  of 
mismatches  to  matches  in  the  data  elements  from  which  notes  were  taken  between 
the  two  sessions.  The  range  across  the  eight  participants  was  a  "low"  of  1.27 
mismatches  for  every  match  to  a  "high"  of  8.50  mismatches  for  every  match.  The 
mean  was  3.73,  an  average  of  about  four  mismatches  for  every  match.  There  was 
thus  a  greater  within-subject  difference  in  note  taking  than  in  element  viewing 
when  measured  at  the  data  element  level  as  the  mean  ratio  across  all  partici¬ 
pants  in  the  elements  viewed  was  1.44  mismatchs  for  every  match. 

Again  as  was  done  with  data  element  viewing  we  compared  the  within-subject 
ratios  to  those  ratios  obtained  by  comparing  each  participant's  note  taking 
with  that  of  every  other  participant  on  the  opposite  scenario.  We  found  that 
over  the  112  comparisons  thus  obtained,  the  within  participant  note  taking 
ratio  was  better  (i.e.,  lower)  than  the  between-subject  measures  92  times 
or.821,  compared  to  the  .723  obtained  for  the  similar  measure  in  element  view¬ 
ing.  Thus  there  appeared  to  be  some  effect  of  individual  differences. 

We  also  compared  the  within-scenario ,  between-subject  mean  ratios  to  the 
between-scenario,  between-subject  mean  ratios  for  the  data  elements  from  which 
notes  were  taken.  Surprisingly,  the  between-scenario  means  were  lower  than  the 
within-scenario  means  in  seven  of  the  16  comparisons,  suggesting  that  the  sce¬ 
nario  effect  on  where  notes  were  taken  was  not  that  great.  Also  there  was  no 
difference  between  students  and  instructors  in  these  consistency  ratios. 

Problem  Solutions 


Participants  were  given  considerable  latitude  as  to  the  amount  of  detail 
they  might  include  in  their  concept  of  operation.  They  were  not  required  to 
complete  all  paragraphs  of  the  operations  estimate,  but  could  determine  for 
themselves  what  represented  adequate  documentation.  Subsequently,  there  was  a 
good  deal  of  individual  difference  in  the  amount  and  type  of  detail  provided. 
The  general  conclusions  drawn  by  the  participants,  however,  were  very  similar 
in  both  scenarios.  In  the  offensive  scenario,  seven  of  the  eight  participants 
decided  to  make  their  main  attack  along  the  northern  of  the  two  axis.  In  the 
defensive  scenario,  seven  of  the  eight  concluded  that  the  enemy  main  attack 
would  cone  along  the  northern  of  the  two  avenues  of  approach.^  However,  there 
was  considerable  variance  in  the  details  used  to  develop  their  concepts  of 
operation. 

In  both  scenarios,  there  were  two  committed  brigades  and  one  reserve  bri¬ 
gade.  In  the  offense  scenario,  five  of  the  participants  changed  the  brigade  at¬ 
tachments  of  maneuver  battalions,  typically  altering  the  structure  of  the 


3 

Although  both  scenarios  took  place  in  West  Germany,  they  were  not  on  the  same 
terrain.  Thus,  there  was  no  carry  over  of  avenue  data  from  one  scenario  to  the 
other . 
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northern  (main  attack)  brigade  and  the  reserve  brigade.  Two  of  the  partici¬ 
pants  selected  one  of  the  two  courses  of  action  suggested  in  the  commander's 
guidance  and  added  little  detail  to  indicate  creative  planning  beyond  that 
point.  The  six  that  showed  more  creative  planning  concentrated  primarily  on 
the  seizing  of  river  crossing  sites  some  ten  kilometers  behind  enemy  lines,  a 
critical  part  of  this  mission,  and  all  but  one  of  them  used  airmobile  force(s) 
to  seize  the  crossings.  Beyond  this,  one  participant  built  a  two  battalion 
task  force  around  the  division  cavalry  squadron  (H-Series  TO&E)  to  take  a 
critical  narrow  avenue  that  begins  the  northern  axis.  One  of  the  instructor 
participants  airlifted  a  battalion  of  the  corps  reserve  brigade  onto  the  north¬ 
ernmost  of  the  three  division  objectives  because  his  wargaming  indicated  that 
sufficient  organic  forces  would  not  exist  to  seize  that  objective. 

In  the  defensive  scenario,  the  primary  decision  was  where  to  locate  the 
reserve  brigade.  Although  all  but  one  placed  the  reserve  in  a  central  loca¬ 
tion,  to  be  able  to  respond  to  either  avenue,  three  placed  the  reserve  well 

forward,  within  15  kilometers  of  the  frontline  while  the  others  had  the  reserve 
19-22  kilometers  back.  Again,  there  was  much  variance  as  to  the  amount  of 
detail  included  in  the  concepts  with  two  participants  adding  little  detail  to 
the  commander's  guidance.  Some  of  the  txtre  creative  solutions  included  the  use 
of  the  division  calvary  squadron  by  two  participants  to  operate  in  an  economy 
of  force  role  in  the  center  to  narrow  the  sectors  of  the  two  committed  brig¬ 
ades.  Another  participant  formed  an  additional  division  reserve  force  to  pro¬ 
tect  the  southern  avenue.  Perhaps  the  most  novel  solution  was  the  use  of  a 

reinforced  division  reserve  by  one  of  the  student  participants  to  attack  the 

enemy  second  echelon  force  to  spoil  the  enemy's  probable  attack.  As  was  also 
true  in  the  offensive  scenario,  several  of  the  participants  went  beyond  their 
division  role  to  reorganize  the  brigade  infrastructures. 

The  student  participants  typically  used  more  novel  and  high  risk  concepts 
than  did  the  instructors.  Although  two  of  the  instructors  added  considerable 
detail  to  their  concepts,  their  approaches  were  more  conservative  and  less 
likely  to  stray  from  the  commander's  guidance  or  the  seeming  practicalities  of 
the  situation.  For  instance,  in  the  defense,  several  students  planned  to  move 
committed  units,  a  risky  undertaking. 

Embedded  Figures  Test  Results 

It  was  hypothesized  that  the  Embedded  Figures  Test  results  would  be  corre¬ 
lated  with  the  number  of  different  information  items  examined  and  the  level  of 
detail  of  the  information  items.  A  high  score  on  the  EFT  indicates  a  global 
orientation  toward  information  processing  and  decision  making,  and  a  low  score 
an  analytic  orientation.  Product  moment  correlations  were  calculated  between 
the  EFT  scores  and  percent  of  detail,  information  items  asked  for,  number  of 
calls,  time  spent  in  alpha  numeric  search,  time  in  graphic  search,  percent  of 
total  time  in  graphic  search  and  total  search  time.  Only  the  relationship 
between  the  EFT  and  total  search  time  was  significant  (r  =  .886,  p  <  .01).  No 
differences  were  found  between  the  EFT  scores  of  the  instructors  and  students. 
However,  a  significant  improvement  was  shown  between  the  first  half  of  the  EFT 
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test  and  the  second  test  (t  =  2.952,  p  <  .05).  This  result  indicates  a  prac¬ 
tice  effect  of  the  EFT  test  and  is  consistent  with  the  findings  of  Within  et. 
al.  (1977). 

These  results  generally  indicate  that  there  is  no  relationship  between 
cognitive  style  as  measured  by  the  EFT  and  pattern  of  information  usage  in  this 
task.  The  one  significant  finding  may  be  an  artifact  of  the  number  of  correla¬ 
tions  that  were  calculated. 

Quality  of  Experiment  Questionnaires 

Table  D-8  shows  the  mean  responses  to  each  Part  I  item  by  scenario.  A  one 
way  analysis  of  variance  (ANOVA)  showed  no  difference  between  instructors  and 
students  in  their  response  to  the  items,  and  a  repeated  measures  ANOVA  showed 
no  significant  differences  between  participant  ratings  of  Scenario  1  and  Sce¬ 
nario  2.  The  mean  response  to  the  items  is  2.5,  somewhat  favorable.  Generally 
the  participants  thought  the  training  session  was  well  organized  and  complete 
and  the  procedure  was  easy  to  use.  However  they  tended  to  disagree  with  the 
statement  that  the  quality  of  their  estimate  in  the  defensive  scenario  was  as 
good  as  it  would  have  been  on  the  battlefield. 

Appendix  E  shows  verbatim  responses  given  by  the  participants  to  the  open 
ended  questions.  The  following  summarizes  the  points  made  by  the  participants 
to  these  questions. 

Experimental  Method:  In  answer  to  the  question,  "Can  realistic  data  on 
information  important  to  tactical  decision  makers  be  gained  with  this  method"? 
19  of  30  questionnaires^  showed  a  "yes"  and  eight  said  "maybe,"  "probably,"  or 
"yes/no."  Two  questionnaires  did  not  answer  this  question,  and  one  said  he 
didn't  know  because  the  experiment  was  a  mental  situation  and  not  a  field 
situation. 

Graphics;  Generally  the  participants  were  not  satisfied  with  the  graphics. 
All  of  the  participants  would  have  preferred  maps  to  the  the  computer  graphics. 
The  most  favorable  comment  with  regard  to  the  graphics  was  that  the  resolution 
of  the  graphics  was  adequate  to  do  the  OPORD  and  that  it  was  of  the  same  qual¬ 
ity  as  TCS/TCT,  (i.e.,  the  terminals  used  in  the  Army's  new  command  and  control 
system,  the  Maneuver  Control  System).  One  participant  thought  the  graphics 
would  be  a  good  addition  to  the  maps  -  but  not  in  lieu  of  the  maps.  Others 
mentioned  it  was  easier  to  use  a  1:50,000  wall  map:  Maps  give  a  clearer  pic¬ 
ture,  and  it  is  hard  to  see  the  big  picture  without  a  map.  One  thought  there 
was  not  enough  detail,  three  that  there  was  too  much  detail,  and  one  that  some¬ 
times  there  was  too  much  and  other  times  not  enough  detail.  One  participant 
said  he  needed  a  pencil  and  paper  to  pictorially  view  the  friendly  and  enemy 
situations.  Other  comments  included  trouble  correlating  data  base  information 

^Questionnaires  from  both  the  first  and  second  sets  of  participants  were  ana¬ 
lyzed.  Although  objective  data  from  the  first  set  was  unusable  due  to  system 
problems,  it  was  felt  that  the  questionnaire  responses,  if  anything,  should  be 
biased  to  the  critical  side  from  this  group. 
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Quality  of  Experiment  Questionnaire 
Mean  Response  to  Items  _  _ 


Defense^ 

Scenario 

Offense^ 

Scenario 

1. 

The  training  session  was  well  organized. 

1.1 

1.0 

2. 

The  training  session  included  everything 

1  needed  to  know. 

1.7 

1.8 

3. 

The  Interactor  in  the  experiment  distracted 
me  from  my  task. 

4.7 

4.8 

4. 

I  would  have  preferred  using  the  computer 
myself . 

4.1 

00 

5. 

The  historical  information  included  every¬ 
thing  I  needed. 

2.2 

2.2 

6. 

The  historical  information  was  well 
organized. 

1.8 

1.6 

7. 

The  reference  information  included  every¬ 
thing  I  needed. 

2.3 

2.0 

8. 

The  reference  information  was  well 
organized. 

1.8 

1.6 

9. 

The  graphics  were  confusing. 

3.0 

2.9 

10. 

The  graphics  were  not  detailed  enough. 

3.9 

3.6 

11. 

The  quality  of  my  estimate  was  as  good  or 
better  than  those  I  would  have  made  on  the 
battlef ield. 

3.3 

2.6 

12. 

I  used  the  same  information  I  would  have 
used  in  a  battlefield  situation. 

2.3 

1.6 

13. 

The  battle  simulation  was  realistic. 

2.1 

2.2 

14. 

The  task  procedure  was  difficult  to  learn. 

4.4 

4.0 

15. 

The  Working  File  was  easy  to  use. 

1.9 

2.0 

Grand  Mean 

2.42 

2.62 

1  1 
5 

2 

=  Strongly  Agree  The  coding  on  items  3,  4,  9, 

=  Strongly  Disagree  been  reversed  for  the  grand 

10,  and  14 
means  only. 

with  the  graphics,  confusing  colors,  the  graphics  were  generally  confusing,  too 
much  screen  clutter,  a  larger  screen  was  needed,  the  graphics  were  too  small, 
the  graphics  were  hard  to  read,  the  unit  boundaries  were  difficult  to  read,  the 
color  scheme  should  correspond  to  that  of  a  military  map  -  canned  map  displays 
were  needed,  and  it  was  difficult  to  go  back  and  forth  across  the  displays. 

Features  or  information  the  participants  thought  the  graphics  should  have 
included  were;  terrain  templating  for  Go,  Slow-Go,  and  No  Go  terrain;  avenue 
of  approach  graphics;  elevations;  highlighting  for  the  borders  and  outlines; 
hilltops;  graphics  by  type  and  size  of  force;  COA  (Course  of  Action)  graphics; 
and  the  ability  to  superimpose  (e.g.,  yesterday's  graphics  on  today's).  One 
said  he  thought  more  training  in  using  the  graphics  would  help,  and  another 
commented  that  a  COA  sketch  is  different  from  an  operation  overlay. 

Three  of  the  sixteen  participants  had  some  degree  of  color  blindness  and 
said  that  they  had  difficulty  differentiating  several  shades  used  in  the  ter¬ 
rain  displays.  Color  deficiency  of  the  user  population  is  an  issue  that  should 
be  addressed  in  the  design  of  the  graphics  display.  Some  form  of  adaptive  color 
graphics  could  be  designed.  For  example,  the  display  might  allow  the  user  to 
change  the  background  color,  change  all  the  greens  to  various  shades  of  blue  if 
he  were  insensitive  to  shades  of  green. 

Scenarios:  Four  of  the  16  participants  commented  that  the  scenarios  were 

not  realistic;  three  of  these  said  the  enemy  force  was  too  large  for  the  offen¬ 
sive  mission,  one  said  the  defense  scenario  was  not  realistic  because  a  linear 
defense  was  depicted  and  one  just  commented  that  more  realistic  scenarios  were 
needed.  The  offensive  situation  was  cited  as  more  difficult  and  challenging. 
Another  participant  said  that  he  had  more  questions  to  ask  of  the  experimenter 
in  the  offensive  scenario. 

Data  Base:  Most  participants  were  generally  satisfied  with  the  textual 
data  base  provided,  however,  a  number  of  additional  items  were  thought  to  be 
needed.  Five  participants  said  combat  (force)  ratios  were  needed.  Other  data 
cited  as  needed  were  the  corps  commander's  intent,  the  previous  operations  plan 
and  operations  estimate,  support  type  missions  for  field  artillery  (GS/GSR/R), 
analysis  of  all  avenues  of  approach  into  the  division  area,  adjacent  unit 
strength,  the  capability  to  task  organize  the  corps  field  artillery  (FA)  into 
division  combat  support  units,  the  course  of  action  the  enemy  favors,  smoke 
generation  units,  more  field  artillery  (including  multiple  rocket  launchers), 
intelligence  collection  assets,  an  expanded  discussion  of  the  courses  of  ac¬ 
tion,  and  percent  fill  of  equipment  and  personnel.  The  intelligence  estimate 
should  have  recommendations  on  courses  of  action.  One  participant  commented 
chat  the  format  of  Che  estimate  was  not  best  for  automated  data  processing. 

With  respect  to  general  comments  about  the  data  base,  five  participants 
felt  there  was  more  information  than  could  be  used;  it  was  difficult  to  process 
all  Che  information  available,  and  the  critical  information  should  have  been 
identified.  One  said  all  the  information  needed  was  available,  it  just  took 
time  to  learn  Che  system.  Two  felt  the  system  could  not  be  used  at  corps  level 
because  the  data  base  was  not  adequate. 
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Realism:  Most  participants  said  that  the  overall  approach  required  to  con¬ 
struct  the  estimate  was  good.  One  commented  it  was  adequate  for  the  commander's 
estimate  and  COA  development.  Six  pointed  out  that  the  situation  was  not  real¬ 
istic  because  the  G3  would  have  been  able  to  discuss  the  COA's  with  his  staff, 
and  get  their  opinions;  this  would  have  helped  him  establish  the  credibility  of 
the  data.  Further,  they  could  have  pressed  their  G2  for  better  estimates  than 
they  had  been  provided.  A  related  point  was  that  more  data  were  needed  on  enemy 
and  friendly  leader  characteristics.  Presumably  some  of  this  would  have  been 
obtained  from  the  staff  interaction.  One  commented  that  the  method  did  not 
allow  for  subjective  reasoning.  Other  comments  include  that  it  would  have  been 
more  appropriate  for  the  decision  maker  to  be  the  commander  rather  than  the  G3, 
and  that  the  G3  should  have  had  the  ability  to  give  missions  to  the  field  ar¬ 
tillery  in  order  to  influence  combat  power. 

Experimental  Task:  One  participant  felt  only  paragraphs  3,  4,  and  5  of  the 
estimates  should  have  been  required.  Two  said  there  was  not  enough  time  for 
the  task,  and  two  others  felt  more  training  should  have  been  given  in  using  the 
data  base  and  graphics. 

In  summary,  most  of  the  participants  felt  the  method  used  in  this  experi¬ 
ment  would  produce  realistic  data  on  information  usage  and  decision  processes 
in  tactical  decision  making.  However,  they  were  not  generally  satisfied  with 
the  graphics  and  had  a  number  of  suggestions  for  improving  the  scenarios,  data 
base,  and  experimental  task. 
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APPENDIX  E 


Verbatim  Responses  to  Quality  of  Experiment  Questionnaire  Open-Ended  Questions 


Questions : 

1.  Describe  any  important  items  of  information  that  were  not  in  the  data  base. 

2.  Do  you  think  realistic  data  on  what  information  is  important  to  tactical 
decision  makers  and  how  they  use  it  can  be  gained  from  using  this  method? 

If  your  answer  is  no,  what  do  you  think  are  the  major  drawbacks? 

3.  General  Comments. 


Pilot  Subject,  Session  1. 

1.  Force  ratios  (very  important)  Z  filled  of  equipment  and  personnel. 

2.  Yes 

3.  Color  scheme  on  map  display  should  correspond  to  military  map.  You  need 
"canned"  map  displays.  Write  only  para  3,4,  and  5  of  estimate.  You  are 
testing  decision  making  but  have  the  "testee”  in  the  role  of  G3  plans. 
Suggest  you  change  role  to  that  of  division  commander. 


Pilot  Subject,  Session  2. 

1.  All  information  was  available.  It  took  time  to  learn  system  on  where  data 
was  located  and  what  and  when  to  ask  for  information.  When  this  informa¬ 
tion  was  asked  for,  it  was  readily  available  through  the  interactor. 

2.  Yes 

3.  The  only  problem  I  faced  was  the  ability  to  readjust  from  asking  for  data 
verbally  from  the  Gl,  G2,  G4  versus  reading  the  information  from  the  data 
base  and  then  having  to  analyze  what  was  there. 


Instructor  1,  Defense  Scenario,  Session  1. 

1.  Could  have  used  a  more  expanded  discussion  on  differences  of  COA's, 
(courses  of  action). 

2.  To  some  degree. 
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3.  The  difficulties  encountered  were  absorbing  all  the  data  quickly  and  being 
able  to  go  back  and  forth  on  the  screen.  Not  having  any  notes  or  paper  is 
foreign  to  my  usual  procedures.  It  takes  some  adjusting  to  be  able  to  use 
all  the  data.  The  difficulty  is  really  picking  the  important  information 
the  first  time  through.  There  was  not  really  a  great  deal  of  differnce 
given  between  each  COA.  Difficult  to  list  many  advantages  and  disadvan¬ 
tages  . 


Instructor  1,  Offense  Scenario,  Session  2. 

1.  The  intel  estimate  did  not  provide  a  recommendation  on  courses  of  action  or 
data  on  force  ratios.  In  real  life,  I  would  have  pressed  the  G2  for  a 
better  estimate  and  a  recommendation. 

2.  Yes 

3.  A  great  deal  of  data  can  be  made  available  including  usage  rates  etc.  All 
this  would  be  important  to  a  planning  staff  in  the  estimate  process.  Time 
available  and  having  only  one  person  led  to  the  exclusion  of  very  detailed 
data  for  this  estimate.  It  would  and  should  be  used  in  a  real  staff 
situation. 


Instructor  2,  Defense  Scenario,  Session  1. 

1.  Included  in  taped  comments. 

2.  Yes 

3.  Re  //ll;  Interface  with  key  personnel  to  include  their  opinions  and  "feel" 
for  the  situation,  their  varying  degrees  of  credibility,  etc.,  would 
enhance  the  decision  making  process  considerably. 


Instructor  2,  Offense  Scenario,  Session  2. 


NO  COMMENTS 


Student  2,  Defense  Scenario,  Session  1. 

1.  Just  hilltops,  rivers  over  certain  width  —  (intelligence  preparation  of 
the  battlefield).  Ability  to  superimpose  (i.e.,  yesterday  situation 
graphically  then  todays).  Graphics  by  type  of  forces  and  size  of 
forces.  COA  graphic. 

2.  Yes 

3.  No  comment 
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Student  2,  Offensive  Scenario,  Session  2. 

1.  Corps  commanders  intent.  True  task  organization  even  those  DS  (direct 
support).  OPLANS  of  units  to  N&S  (north  and  south)  of  Div. 

2.  Yes 

3.  Outstanding.  I  learned  more  about  ny  own  information  requirements  to  make 
a  decision  which  will  facilitate  decision  making  in  the  future. 


Student  1,  Defensive  Scenario,  Session  1. 

1.  GS/GSR/R  (genreal  support/general  support  reinforcing/reinforcing)  missions 
for  FA  (field  artillery)  -  only  the  DS  mission  can  be  applied.  Need  air 
avenue  of  approach  graphics.  Ocher  2  types  of  air  graphics  are  not  used 

in  GOA  or  OPORD  overlay  graphics. 

2.  Yes 

3.  System  is  interesting,  easy  to  use,  and  adequate  for  Cdrs  estimates  and  COA 

development.  Resolution  of  graphics  is  enough  to  do  opord  graphics  of  the 

same  quality  (or  better)  as  TCS/TCT.  A  good  system  and  interesting 

experiment. 


Student  1,  Offensive  Scenario,  Session  2. 

1.  No  comment 

2.  Yes 

3.  Tactical  situation  for  second  situation  more  difficult.  More  challeng¬ 
ing.  I  find  the  experiment  and  its  objectives  interesting. 

Instructor  3,  Offensive  Scenario,  Session  1. 

1.  The  previous  operations  estimate  to  use  for  a  boilerplate  -  how  informa 
tion  would  be  presented. 

2.  No  comment 

3.  The  graphics  would  be  a  good  addition  to  use  of  a  printed  map  but  not  in 
lieu  of  having  a  map  for  reference.  A  more  realistic  scenario  would  be 
easier  to  plan  a  reasonable  course  of  action. 
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Instructor  3,  Defensive  Scenario,  Session  2. 

1.  Combat  rations,  elevations,  previous  OPLAN  and  OPS  estimate. 

2.  Yes 


3.  With  expansion  this  seems  to  be  a  good  approach  to  an  aid  to  the  decision 
making  process.  A  staff  would  need  training  and  experience  with  such  a 
system  to  be  effective.  A  very  good  training  tool.  Would  like  to  use 
as  part  of  the  CAS'^  (Combined  Arms  and  Services  Staff  School)  program. 


Instructor  4,  Defensive  Scenario,  Session  2.  (Single  questionnaire) 

1.  Intelligence  preparation  of  the  battlefield.  Artillery  combat  mission. 
Participant  is  also  forced  to  make  some  assumptions  that  would  be 
routinely  provided  by  other  staff  officers. 

2.  Probably 

3.  However,  the  organization  of  data  needs  work  and  the  ability  to  use  the  map 
display  is  limited  by  the  small  size  of  the  display. 


Student  3,  Offensive  Scenario,  Session  1. 

1.  !1I  (military  intelligence)  collection  assets. 

2.  Yes 

3.  I  think  that  the  use  of  a  map  as  a  commander's  tool  (rather  than,  or  at 
worst  in  addition  to,  the  graphics  terminal)  would  ease  the  players 
initial  interaction  with  the  scenario.  If  the  intent  is  to  use  all 

the  automated  devices  -  terminal  as  well  as  graphic  display  -  the  set 
up  scenario  is  fine.  If  the  intent  is  to  develop  the  true  tools 
used  by  the  Cdr  to  develop  a  CA  (course  of  action),  then  the  use  of 
the  map  may  give  a  clearer  picture  (using  a  known  vehicle  reduces 
the  need  to  learn  a  new  one).  Very  enjoyable.  The  facilitator  and 
the  interactor  minimized  the  anxiety  and  eased  the  process. 


Student  3,  Defensive  Scenario,  Session  2. 

1.  Historical  Information:  GSR  Status  of  ARTY. 

2.  Yes 

3.  I  sure  wish  I'd  have  had  more  F  or  some  MRLs ,  (multiple  rocket  launchers). 
Attrition  factors  for  friendly  was  probably  accurate;  I  hope  the  strength 
factors  for  enemy  were  high. 
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Student  4,  Offensive  Scenario,  Session  1. 


1.  Snoke  generator  units  -  very  important  to  river  crossing  operations. 

2.  Yes 

3.  It  was  hard  trying  to  see  the  big  picture  when  studying  the  estimates  with¬ 
out  a  map  to  work  on.  The  loss  of  pencil  and  paper  prohibited  me  from  pic- 
torically  visualizing  the  friendly  and  enemy  situation.  The  visual 
assist  was  very  confusing.  The  level  of  resolution  was  almost  too 
detailed.  I  would  much  rather  see  simple  graphics  and  not  so  many  on 

the  screen  at  any  one  time.  My  COA  sketch  should  look  like  this: 

Howver  a  COA  sketch  is  different  throw  an  operation  overlay.  (COA 
SKETCH  drawn) 


Student  4,  Defensive  Scenario,  Session  2. 

1.  No  comment 

2.  No  comment 

3.  Overall  very  good.  Personnel  and  Log  estimites  recommended  COA's  which 
they  favored;  in  defense,  we  should  look  at  COA's  that  the  enemy  favors. 


Instructor  5,  Defensive  &  Offensive  Scenarios,  Session  1  &  2.  (Single 
questionnaire) 

1.  Combat  power  ratio  determination  sheet/outline. 

2.  Yes/No.  At  corps  level  the  use  of  1:250,000  is  SOP  (standard  operating 
procedure).  At  division,  1:50,000.  You  cannot  see  the  entire  battlefield 
on  visual  using  1:50,000.  Hard  to  track.  Maps  much  better.  Storage  of 
historical  data  and  estimate  data  would  be  a  great  help  at  all  levels. 


Instructor  6,  Defensive  Scenario,  Session  1. 

1.  Ability  to  give  FA  missions  to  influence  combat  power, 

2.  Some.  Estimates  at  div  level  will  most  likely  be  oral. 

3.  Most  applicable  at  div  for  OPLAN  estimates  due  to  great  detail.  Probably 
most  usable  at  corps  level  procedurally  but  data  base  is  not  currently 
sufficient . 
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Instructor  6,  Offensive  Scenario,  Session  2. 

1.  No  comment 

2.  Maybe 

3.  Not  enough  detail  on  map/overlay.  Combat  ratio  algorithms  need  to  be  in 
computer. 

Student  5,  Defensive  Scenario,  Session  1. 

1.  Refer  to  interactors  (observer's  RM)  notes. 

2.  Maybe 

3.  Need  to  have  more  data  on  human  factors  -  leaders  characteristics,  etc. 


Student  5,  Offensive  Scenario,  Session  2. 

1.  Interactor  has  my  notes  on  type  and  quality  of  information  that  should  be 
included  to  make  this  more  realistic. 

2.  Yes.  If  the  information  is  realistic,  e.g.,  equipment  strengths,  are 
significantly  different  etc. 


Student  6,  Defensive  Scenario,  Session  1. 

1.  Need  to  be  able  to  task  organize  corps  FA  into  the  division  combat  support 
assets.  Need  to  look  at  corps  engineers  too.  This  should  be  task  organized 
into  the  division  CS  (combat  support)  assets  -  or  at  least  have  the  capa¬ 
bility  to  do  so. 

2.  Yes 

3.  I  believe  the  problem  is  not  too  little  information  but  too  much.  The  com¬ 
mander  needs  selective  info  to  make  decisions.  You  must  identify  which 
information  is  critical. 

Student  6,  Offensive  Scenario,  Session  2. 

1.  Corps  FA  and  engineers  could  not  be  task  organized  into  div  task  organize 
tion. 

2.  Yes 

3.  Only  problem  was  enemy  force.  Much  too  large  for  div  mission.  Other  than 
that,  no  significant  problems. 


E-6 


Instructor  1,  Offensive  Scenario,  Session  1. 

1.  Have  been  with  J  series  (Table  of  Organization  and  Equipment  -  TO&E)  since 
1980.  Hard  to  digress  back  to  H-series. 

2.  Data  -  "Yes".  "No  Staff"  to  answer  questions  pertaining  to  data. 

3.  Fun  stuff. 

Instructor  1,  Defensive  Scenario,  Session  2. 

1.  Again,  larger  screen  is  needed. 

2.  Yes  small  graphics. 

3.  Had  fun.  Enjoyed  working  with  Rex  and  Glen. 

Instructor  8,  Offensive  Scenario,  Session  1. 

1.  Analysis  of  all  avenues  of  approach  to  the  div  area. 

2.  Yes 

3.  No  comment 

Instructor  8,  Defensive  Scenario,  Session  2. 

1.  No  comment 

2.  Yes 

3 .  No  comment 

Student  1,  Offensive  Scenario,  Session  1. 

1.  No  comment 

2.  Yes 

3.  Reserved  for  Now. 
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Student  1,  Defensive  Scenario,  Session  2. 

1.  3  GE  (German)  corps,  28  Panzer  division  strength. 

2.  ?  1  don't  know  because  this  is  a  mental  environment  -  not  -  a  field  situa 
tion. 

3.  I  had  more  questions  that  couldn't  be  answered  on  the  offense  rather  than 
defense.  It  may  be  personal,  or  it  may  have  been  because  the  offense 
requires  more  information  (more  unknowns).  Therefore,  this  method  may 

be  more  applicable  to  defense,  rather  than  offense.  I  didn't  like  the 
computer  maps  (too  constricted).  Would  rather  use  large  relief  maps  that 
I'm  used  to.  Method  didn't  allow  for  more  subjective  (gut  feel)  reasoning 
Also,  the  environment  was  isolated,  so  logical  assumptions  used  may  not 
have  been  valid,  because  could  not  be  discussed  with  colleagues  (even  in 
face  of  facts  presented). 


Student  8,  Offensive  Scenario,  Session  1. 

1.  Corps  units  assigned  to  support  division  were  not  on  division  troop  list 
and  therefore  could  not  be  task  organized  with  the  brigades.  No  terrain 
templating  was  available  in  data  base  to  portray  GO,  SLOW-GO,  NO-GO 
terrain. 

2.  Yes 

3.  16th  Kech  Div  faced  too  strong  an  enemy  force  to  accomplish  its  mission. 


Student  8,  Defensive  Scenario,  Session  2. 

1.  No  comment 

2.  Yes 

3.  No  comment 
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